2023-12-01T16:43:18.263 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-01T16:43:18.264 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838 --owner scheduled_yuriw@teuthology --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} --name yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/orig.config.yaml 2023-12-01T16:43:18.306 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T16:43:18.387 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838 branch: wip-neorados-learning-from-experience description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7473838' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-neorados-learning-from-experience ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: d710c09d2ffe0f0858468330fd3d69f1f4506222 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: d710c09d2ffe0f0858468330fd3d69f1f4506222 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-neorados-learning-from-experience sha1: d710c09d2ffe0f0858468330fd3d69f1f4506222 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 5183 sha1: d710c09d2ffe0f0858468330fd3d69f1f4506222 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-neorados-learning-from-experience suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d710c09d2ffe0f0858468330fd3d69f1f4506222 targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBQIyZoIHRylwtXjzYm+hE6V8vXmABu3fn7ivKqnzNMIizqP9FIVhl3+6s6v1db7RCg8qcA69TIPmuMquZ12nzg= smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsY9SWMXphNX4his3hpk3d0Ii08nhsOTfGtAq7TpsCy1q9isqLbgjSEt+no/DLci5rxXXU1Cwoatw1jkwPA7Qw= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-01_15:09:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-01T16:43:18.387 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa; will attempt to use it 2023-12-01T16:43:18.388 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa/tasks 2023-12-01T16:43:18.388 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-01T16:43:18.390 INFO:teuthology.task.internal:Checking packages... 2023-12-01T16:43:18.415 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd710c09d2ffe0f0858468330fd3d69f1f4506222' 2023-12-01T16:43:18.415 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-01T16:43:18.415 INFO:teuthology.packaging:ref: None 2023-12-01T16:43:18.416 INFO:teuthology.packaging:tag: None 2023-12-01T16:43:18.416 INFO:teuthology.packaging:branch: wip-neorados-learning-from-experience 2023-12-01T16:43:18.416 INFO:teuthology.packaging:sha1: d710c09d2ffe0f0858468330fd3d69f1f4506222 2023-12-01T16:43:18.416 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-neorados-learning-from-experience 2023-12-01T16:43:18.628 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7570.gd710c09d 2023-12-01T16:43:18.629 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-01T16:43:18.635 INFO:teuthology.task.internal:no buildpackages task found 2023-12-01T16:43:18.635 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-01T16:43:18.650 INFO:teuthology.task.internal:Saving configuration 2023-12-01T16:43:18.666 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-01T16:43:18.672 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T16:43:18.699 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 16:38:16.157771', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBQIyZoIHRylwtXjzYm+hE6V8vXmABu3fn7ivKqnzNMIizqP9FIVhl3+6s6v1db7RCg8qcA69TIPmuMquZ12nzg='} 2023-12-01T16:43:18.723 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 16:38:16.155820', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsY9SWMXphNX4his3hpk3d0Ii08nhsOTfGtAq7TpsCy1q9isqLbgjSEt+no/DLci5rxXXU1Cwoatw1jkwPA7Qw='} 2023-12-01T16:43:18.723 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-01T16:43:18.729 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-12-01T16:43:18.729 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-12-01T16:43:18.729 INFO:teuthology.run_tasks:Running task console_log... 2023-12-01T16:43:18.801 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f531eb19ee0>, signals=[15]) 2023-12-01T16:43:18.802 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-01T16:43:18.807 INFO:teuthology.task.internal:Opening connections... 2023-12-01T16:43:18.807 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T16:43:18.809 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:43:18.908 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T16:43:18.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:43:18.991 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-01T16:43:18.997 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2023-12-01T16:43:19.015 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2023-12-01T16:43:19.015 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-01T16:43:19.073 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-01T16:43:19.074 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-01T16:43:19.074 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2023-12-01T16:43:19.102 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2023-12-01T16:43:19.121 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2023-12-01T16:43:19.122 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2023-12-01T16:43:19.175 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2023-12-01T16:43:19.175 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2023-12-01T16:43:19.175 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-12-01T16:43:19.176 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-01T16:43:19.176 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2023-12-01T16:43:19.193 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-01T16:43:19.202 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-01T16:43:19.207 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-01T16:43:19.207 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2023-12-01T16:43:19.210 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2023-12-01T16:43:19.227 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-01T16:43:19.233 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-01T16:43:19.233 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2023-12-01T16:43:19.268 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2023-12-01T16:43:19.547 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-01T16:43:19.596 INFO:teuthology.run_tasks:Running task kernel... 2023-12-01T16:43:19.611 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-01T16:43:19.611 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-01T16:43:19.611 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-01T16:43:19.612 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-01T16:43:19.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T16:43:19.628 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-12-01T16:43:19.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T16:43:19.629 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2023-12-01T16:43:19.682 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-361.el9.x86_64 2023-12-01T16:43:19.683 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-361.el9.x86_64 2023-12-01T16:43:19.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2023-12-01T16:43:19.687 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-361.el9.x86_64 2023-12-01T16:43:19.687 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-361.el9.x86_64 2023-12-01T16:43:19.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2023-12-01T16:43:21.704 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 15 MB/s | 6.4 MB 00:00 2023-12-01T16:43:21.807 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 12 MB/s | 6.4 MB 00:00 2023-12-01T16:43:23.088 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2023-12-01T16:43:23.199 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2023-12-01T16:43:24.020 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 7.9 MB 00:00 2023-12-01T16:43:24.339 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 7.9 MB 00:00 2023-12-01T16:43:26.340 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 20 MB/s | 18 MB 00:00 2023-12-01T16:43:27.030 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 9.7 MB/s | 18 MB 00:01 2023-12-01T16:43:30.748 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 5.8 MB/s | 5.8 MB 00:01 2023-12-01T16:43:31.622 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 5.8 MB 00:01 2023-12-01T16:43:32.033 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 15 kB 00:00 2023-12-01T16:43:33.306 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 27 kB/s | 15 kB 00:00 2023-12-01T16:43:34.436 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 8.7 MB/s | 20 MB 00:02 2023-12-01T16:43:39.001 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 20 MB 00:05 2023-12-01T16:43:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2023-12-01T16:43:41.523 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-01T16:43:41.523 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-01T16:43:41.624 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:43:41.636 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:43:41.637 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:43:41.638 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2023-12-01T16:43:41.638 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:43:41.638 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 96 M 2023-12-01T16:43:41.638 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 121 M 2023-12-01T16:43:41.638 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:43:43.459 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 3.4 MB/s | 5.6 MB 00:01 2023-12-01T16:43:43.881 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2023-12-01T16:43:44.467 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 7.6 MB/s | 20 MB 00:02 2023-12-01T16:43:46.159 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 8.7 MB/s | 38 MB 00:04 2023-12-01T16:43:46.288 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2023-12-01T16:43:46.288 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2023-12-01T16:43:46.379 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:43:46.388 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:43:46.388 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2023-12-01T16:43:46.388 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:43:46.388 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-388.el9 baseos 5.6 M 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core x86_64 5.14.0-388.el9 baseos 20 M 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules x86_64 5.14.0-388.el9 baseos 38 M 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core x86_64 5.14.0-388.el9 baseos 32 M 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout:Install 4 Packages 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:43:46.389 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 96 M 2023-12-01T16:43:46.390 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 121 M 2023-12-01T16:43:46.393 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:43:46.762 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 9.8 MB/s | 32 MB 00:03 2023-12-01T16:43:46.762 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:43:46.762 INFO:teuthology.orchestra.run.smithi092.stdout:Total 19 MB/s | 96 MB 00:05 2023-12-01T16:43:47.610 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:43:47.968 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:43:47.968 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:43:48.066 INFO:teuthology.orchestra.run.smithi067.stdout:(1/4): kernel-5.14.0-388.el9.x86_64.rpm 4.1 MB/s | 5.6 MB 00:01 2023-12-01T16:43:49.290 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:43:49.291 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:43:49.358 INFO:teuthology.orchestra.run.smithi067.stdout:(2/4): kernel-core-5.14.0-388.el9.x86_64.rpm 7.7 MB/s | 20 MB 00:02 2023-12-01T16:43:50.275 INFO:teuthology.orchestra.run.smithi067.stdout:(3/4): kernel-modules-5.14.0-388.el9.x86_64.rpm 11 MB/s | 38 MB 00:03 2023-12-01T16:43:50.575 INFO:teuthology.orchestra.run.smithi067.stdout:(4/4): kernel-modules-core-5.14.0-388.el9.x86_6 13 MB/s | 32 MB 00:02 2023-12-01T16:43:50.575 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:43:50.575 INFO:teuthology.orchestra.run.smithi067.stdout:Total 23 MB/s | 96 MB 00:04 2023-12-01T16:43:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:43:51.893 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:43:51.893 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:43:51.929 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:43:52.565 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-01T16:43:52.700 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-01T16:43:53.255 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:43:53.255 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:43:55.477 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-01T16:43:55.703 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-01T16:43:56.043 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:43:56.708 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-core-5.14.0-388.el9.x86_64 1/4 2023-12-01T16:43:56.833 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-01T16:43:59.727 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-01T16:43:59.901 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-01T16:44:00.302 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-01T16:44:00.418 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:04.874 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-01T16:44:05.003 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : kernel-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:05.066 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:10.016 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:33.734 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:33.776 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:34.156 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:34.156 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-01T16:44:34.156 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-01T16:44:34.156 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:44:34.562 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:44:35.054 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T16:44:35.212 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-01T16:44:35.212 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-01T16:44:35.213 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-01T16:44:35.213 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-01T16:44:35.213 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-01T16:44:35.213 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-12-01T16:44:35.213 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2023-12-01T16:44:35.213 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T16:44:35.332 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-01T16:44:35.332 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-01T16:44:35.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2023-12-01T16:44:35.366 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:44:35.367 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2023-12-01T16:44:35.368 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-01T16:44:35.368 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-388.el9.x86_64 2023-12-01T16:44:35.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-01T16:44:35.978 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2023-12-01T16:44:37.362 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-01T16:44:37.398 INFO:teuthology.orchestra.run.smithi092.stderr:done 2023-12-01T16:44:37.400 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2023-12-01T16:44:37.422 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.r3aODjFK6k 2023-12-01T16:44:37.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.r3aODjFK6k 2023-12-01T16:44:37.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.r3aODjFK6k 2023-12-01T16:44:37.626 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.r3aODjFK6k is 6KB 2023-12-01T16:44:37.675 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.r3aODjFK6k 2023-12-01T16:44:37.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2023-12-01T16:44:37.759 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-01T16:44:37.760 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-01T16:44:37.760 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-01T16:44:37.760 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-01T16:44:37.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-01T16:44:37.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2023-12-01T16:44:40.470 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-core-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:40.515 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-modules-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-388.el9.x86_64 1/4 2023-12-01T16:44:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-core-5.14.0-388.el9.x86_64 2/4 2023-12-01T16:44:40.922 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-5.14.0-388.el9.x86_64 3/4 2023-12-01T16:44:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-modules-core-5.14.0-388.el9.x86_64 4/4 2023-12-01T16:44:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:44:41.351 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:44:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-388.el9.x86_64 2023-12-01T16:44:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-core-5.14.0-388.el9.x86_64 2023-12-01T16:44:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-5.14.0-388.el9.x86_64 2023-12-01T16:44:41.351 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-modules-core-5.14.0-388.el9.x86_64 2023-12-01T16:44:41.352 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:44:41.352 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:44:41.816 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T16:44:41.969 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-01T16:44:41.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-01T16:44:41.969 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-388.el9.x86_64 2023-12-01T16:44:41.969 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-01T16:44:41.970 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-01T16:44:41.970 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-12-01T16:44:41.970 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2023-12-01T16:44:41.970 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2023-12-01T16:44:42.092 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-388.el9.x86_64 2023-12-01T16:44:42.093 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-388.el9.x86_64 2023-12-01T16:44:42.093 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2023-12-01T16:44:42.134 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2023-12-01T16:44:42.134 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2023-12-01T16:44:42.134 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.06 2023-12-01T16:44:42.134 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 68.el9 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Group : Unspecified 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 8264254 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : builder@centos.org 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : CentOS 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-01T16:44:42.135 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:44:42.136 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2023-12-01T16:44:42.137 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-01T16:44:42.137 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 5.14.0-388.el9.x86_64 2023-12-01T16:44:42.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-01T16:44:42.772 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2023-12-01T16:44:44.141 INFO:teuthology.orchestra.run.smithi067.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-12-01T16:44:44.176 INFO:teuthology.orchestra.run.smithi067.stderr:done 2023-12-01T16:44:44.178 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2023-12-01T16:44:44.194 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.8gaTJR6fBn 2023-12-01T16:44:44.194 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8gaTJR6fBn 2023-12-01T16:44:44.266 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.8gaTJR6fBn 2023-12-01T16:44:44.421 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.8gaTJR6fBn is 6KB 2023-12-01T16:44:44.469 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.8gaTJR6fBn 2023-12-01T16:44:44.484 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2023-12-01T16:44:44.564 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-12-01T16:44:44.564 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-12-01T16:44:44.564 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-12-01T16:44:44.564 INFO:teuthology.orchestra.run.smithi067.stdout:15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64.conf 2023-12-01T16:44:44.565 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-388.el9.x86_64 2023-12-01T16:44:44.718 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2023-12-01T16:45:07.931 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-12-01T16:45:07.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-12-01T16:45:07.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:45:14.726 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-12-01T16:45:14.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-12-01T16:45:14.727 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:45:49.282 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2023-12-01T16:45:58.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-12-01T16:45:58.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:46:07.944 DEBUG:teuthology.orchestra.remote:timed out 2023-12-01T16:46:16.674 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2023-12-01T16:46:16.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-12-01T16:46:16.947 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:46:17.310 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-12-01T16:46:17.895 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-12-01T16:46:17.895 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-388.el9.x86_64"... 2023-12-01T16:46:17.896 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-12-01T16:46:17.910 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-388.el9.x86_64 2023-12-01T16:46:17.910 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-01T16:46:17.910 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-01T16:46:17.910 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-01T16:46:18.911 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-12-01T16:46:18.912 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-01T16:46:18.979 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2023-12-01T16:46:19.013 DEBUG:teuthology.parallel:result is None 2023-12-01T16:46:28.675 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-12-01T16:46:28.676 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:46:29.872 DEBUG:teuthology.orchestra.run.smithi067:> true 2023-12-01T16:46:30.332 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-12-01T16:46:30.332 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-388.el9.x86_64"... 2023-12-01T16:46:30.332 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2023-12-01T16:46:30.347 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-388.el9.x86_64 2023-12-01T16:46:30.347 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-388.el9.x86_64 vs 5.14.0-388.el9.x86_64 2023-12-01T16:46:30.347 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-01T16:46:30.347 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-01T16:46:31.348 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-12-01T16:46:31.348 DEBUG:teuthology.orchestra.run.smithi067:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-01T16:46:31.426 INFO:teuthology.orchestra.run.smithi067.stdout:ttyS1 2023-12-01T16:46:31.460 DEBUG:teuthology.parallel:result is None 2023-12-01T16:46:31.460 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-01T16:46:31.468 INFO:teuthology.task.internal:Creating test directory... 2023-12-01T16:46:31.468 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-01T16:46:31.470 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-01T16:46:31.489 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-01T16:46:31.496 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-01T16:46:31.503 INFO:teuthology.task.internal:Creating archive directory... 2023-12-01T16:46:31.504 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-01T16:46:31.526 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-01T16:46:31.561 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-01T16:46:31.569 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-01T16:46:31.569 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-01T16:46:31.589 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-01T16:46:31.624 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T16:46:31.628 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T16:46:31.641 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T16:46:31.641 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-01T16:46:31.643 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-01T16:46:31.651 INFO:teuthology.task.internal:Configuring sudo... 2023-12-01T16:46:31.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-01T16:46:31.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-01T16:46:31.714 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-01T16:46:31.724 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-01T16:46:31.724 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-01T16:46:31.757 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-01T16:46:31.772 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T16:46:31.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T16:46:31.919 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T16:46:31.962 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T16:46:32.006 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T16:46:32.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-01T16:46:32.079 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T16:46:32.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-01T16:46:32.180 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T16:46:32.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-01T16:46:32.313 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T16:46:32.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-01T16:46:32.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2023-12-01T16:46:32.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2023-12-01T16:46:32.433 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T16:46:32.483 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T16:46:32.927 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-01T16:46:32.934 INFO:teuthology.task.internal:Starting timer... 2023-12-01T16:46:32.935 INFO:teuthology.run_tasks:Running task pcp... 2023-12-01T16:46:32.953 INFO:teuthology.run_tasks:Running task selinux... 2023-12-01T16:46:32.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2023-12-01T16:46:33.030 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2023-12-01T16:46:33.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2023-12-01T16:46:33.081 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2023-12-01T16:46:33.083 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-01T16:46:33.083 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2023-12-01T16:46:33.098 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2023-12-01T16:46:33.098 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2023-12-01T16:46:33.132 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2023-12-01T16:46:33.132 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi092.front.sepia.ceph.com': 'permissive'} 2023-12-01T16:46:33.132 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T16:46:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T16:46:33.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T16:46:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T16:46:33.202 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-01T16:46:33.202 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2023-12-01T16:46:33.241 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2023-12-01T16:46:33.275 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-01T16:46:33.285 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-01T16:46:33.285 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-01T16:46:33.305 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-01T16:46:33.306 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi092.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-01T16:52:20.120 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi092.front.sepia.ceph.com')] 2023-12-01T16:52:20.121 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-12-01T16:52:20.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:52:20.219 DEBUG:teuthology.orchestra.run.smithi067:> true 2023-12-01T16:52:20.290 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-12-01T16:52:20.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-12-01T16:52:20.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-01T16:52:20.374 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-12-01T16:52:20.442 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-12-01T16:52:20.442 INFO:teuthology.run_tasks:Running task clock... 2023-12-01T16:52:20.455 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-01T16:52:20.455 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-01T16:52:20.456 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T16:52:20.458 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-01T16:52:20.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T16:52:20.492 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-01T16:52:20.513 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-01T16:52:20.515 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-01T16:52:20.532 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-01T16:52:20.549 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2023-12-01T16:52:20.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2023-12-01T16:52:20.567 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2023-12-01T16:52:20.574 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2023-12-01T16:52:20.586 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-01T16:52:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-01T16:52:20.606 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-01T16:52:20.607 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-01T16:52:20.651 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2023-12-01T16:52:20.652 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T16:52:20.652 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-12-01T16:52:20.653 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.653 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.653 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.653 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.653 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2023-12-01T16:52:20.656 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T16:52:20.656 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2023-12-01T16:52:20.657 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.657 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.657 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.657 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T16:52:20.658 INFO:teuthology.run_tasks:Running task install... 2023-12-01T16:52:20.665 DEBUG:teuthology.task.install:project ceph 2023-12-01T16:52:20.665 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222'}} 2023-12-01T16:52:20.665 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222'} 2023-12-01T16:52:20.665 INFO:teuthology.task.install:Using flavor: default 2023-12-01T16:52:20.671 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-12-01T16:52:20.672 INFO:teuthology.task.install:extra packages: [] 2023-12-01T16:52:20.672 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222', 'tag': None, 'wait_for_package': False} 2023-12-01T16:52:20.672 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d710c09d2ffe0f0858468330fd3d69f1f4506222 2023-12-01T16:52:20.674 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222', 'tag': None, 'wait_for_package': False} 2023-12-01T16:52:20.674 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d710c09d2ffe0f0858468330fd3d69f1f4506222 2023-12-01T16:52:20.839 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/ 2023-12-01T16:52:20.839 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7570.gd710c09d 2023-12-01T16:52:20.933 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/ 2023-12-01T16:52:20.933 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7570.gd710c09d 2023-12-01T16:52:20.949 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-01T16:52:20.950 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T16:52:20.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-01T16:52:20.977 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-01T16:52:20.977 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d710c09d2ffe0f0858468330fd3d69f1f4506222/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-01T16:52:21.044 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-neorados-learning-from-experience/d710c09d2ffe0f0858468330fd3d69f1f4506222/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-01T16:52:21.044 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T16:52:21.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-01T16:52:21.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-01T16:52:21.077 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-12-01T16:52:21.078 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d710c09d2ffe0f0858468330fd3d69f1f4506222/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-01T16:52:21.125 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-01T16:52:21.151 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-01T16:52:21.190 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2023-12-01T16:52:21.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2023-12-01T16:52:21.239 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-01T16:52:21.269 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2023-12-01T16:52:21.271 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2023-12-01T16:52:21.432 INFO:teuthology.orchestra.run.smithi092.stdout:73 files removed 2023-12-01T16:52:21.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw 2023-12-01T16:52:21.526 INFO:teuthology.orchestra.run.smithi067.stdout:73 files removed 2023-12-01T16:52:21.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw 2023-12-01T16:52:22.152 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 209 kB/s | 79 kB 00:00 2023-12-01T16:52:22.302 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 193 kB/s | 79 kB 00:00 2023-12-01T16:52:22.570 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 26 kB/s | 9.5 kB 00:00 2023-12-01T16:52:22.744 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 24 kB/s | 9.5 kB 00:00 2023-12-01T16:52:22.954 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 5.4 kB/s | 1.9 kB 00:00 2023-12-01T16:52:23.145 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 5.2 kB/s | 1.9 kB 00:00 2023-12-01T16:52:23.373 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-01T16:52:23.555 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - AppStream 17 MB/s | 6.4 MB 00:00 2023-12-01T16:52:24.740 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2023-12-01T16:52:25.006 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2023-12-01T16:52:25.955 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 7.9 MB 00:00 2023-12-01T16:52:26.931 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 7.9 MB 00:01 2023-12-01T16:52:28.865 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 22 MB/s | 18 MB 00:00 2023-12-01T16:52:33.373 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 5.8 MB 00:01 2023-12-01T16:52:33.572 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 18 MB 00:06 2023-12-01T16:52:34.941 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 15 kB 00:00 2023-12-01T16:52:37.095 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 20 MB 00:02 2023-12-01T16:52:37.606 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 12 MB/s | 5.8 MB 00:00 2023-12-01T16:52:39.916 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 15 kB 00:01 2023-12-01T16:52:40.942 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 23 MB/s | 20 MB 00:00 2023-12-01T16:52:41.709 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2023-12-01T16:52:44.374 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:52:44.382 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:52:44.382 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:52:44.382 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:52:44.382 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:52:44.382 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 8.7 M 2023-12-01T16:52:44.382 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 3.4 M 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 3.1 M 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 5.2 M 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 19 M 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 24 k 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 742 k 2023-12-01T16:52:44.383 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 498 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 4.8 M 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 45 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 129 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 161 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-01T16:52:44.384 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 320 k 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 298 k 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 99 k 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout:Install 27 Packages 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 2 Packages 2023-12-01T16:52:44.385 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:52:44.386 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 55 M 2023-12-01T16:52:44.386 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:52:45.526 INFO:teuthology.orchestra.run.smithi092.stdout:(1/29): ceph-radosgw-18.0.0-7570.gd710c09d.el9. 8.3 MB/s | 8.7 MB 00:01 2023-12-01T16:52:45.560 INFO:teuthology.orchestra.run.smithi092.stdout:(2/29): ceph-selinux-18.0.0-7570.gd710c09d.el9. 728 kB/s | 24 kB 00:00 2023-12-01T16:52:45.702 INFO:teuthology.orchestra.run.smithi092.stdout:(3/29): ceph-base-18.0.0-7570.gd710c09d.el9.x86 4.3 MB/s | 5.2 MB 00:01 2023-12-01T16:52:45.727 INFO:teuthology.orchestra.run.smithi092.stdout:(4/29): libcephfs2-18.0.0-7570.gd710c09d.el9.x8 4.3 MB/s | 742 kB 00:00 2023-12-01T16:52:45.844 INFO:teuthology.orchestra.run.smithi092.stdout:(5/29): libradosstriper1-18.0.0-7570.gd710c09d. 3.4 MB/s | 498 kB 00:00 2023-12-01T16:52:45.900 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2023-12-01T16:52:45.944 INFO:teuthology.orchestra.run.smithi092.stdout:(6/29): python3-ceph-argparse-18.0.0-7570.gd710 452 kB/s | 45 kB 00:00 2023-12-01T16:52:46.053 INFO:teuthology.orchestra.run.smithi092.stdout:(7/29): python3-ceph-common-18.0.0-7570.gd710c0 1.2 MB/s | 129 kB 00:00 2023-12-01T16:52:46.245 INFO:teuthology.orchestra.run.smithi092.stdout:(8/29): ceph-common-18.0.0-7570.gd710c09d.el9.x 11 MB/s | 19 MB 00:01 2023-12-01T16:52:46.270 INFO:teuthology.orchestra.run.smithi092.stdout:(9/29): python3-cephfs-18.0.0-7570.gd710c09d.el 744 kB/s | 161 kB 00:00 2023-12-01T16:52:46.296 INFO:teuthology.orchestra.run.smithi092.stdout:(10/29): python3-rados-18.0.0-7570.gd710c09d.el 6.2 MB/s | 320 kB 00:00 2023-12-01T16:52:46.372 INFO:teuthology.orchestra.run.smithi092.stdout:(11/29): librgw2-18.0.0-7570.gd710c09d.el9.x86_ 7.4 MB/s | 4.8 MB 00:00 2023-12-01T16:52:46.398 INFO:teuthology.orchestra.run.smithi092.stdout:(12/29): python3-rgw-18.0.0-7570.gd710c09d.el9. 975 kB/s | 99 kB 00:00 2023-12-01T16:52:46.423 INFO:teuthology.orchestra.run.smithi092.stdout:(13/29): python3-rbd-18.0.0-7570.gd710c09d.el9. 1.9 MB/s | 298 kB 00:00 2023-12-01T16:52:46.548 INFO:teuthology.orchestra.run.smithi092.stdout:(14/29): boost-program-options-1.75.0-8.el9.x86 603 kB/s | 106 kB 00:00 2023-12-01T16:52:46.565 INFO:teuthology.orchestra.run.smithi092.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 950 kB/s | 159 kB 00:00 2023-12-01T16:52:46.591 INFO:teuthology.orchestra.run.smithi092.stdout:(16/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 371 kB/s | 44 kB 00:00 2023-12-01T16:52:46.633 INFO:teuthology.orchestra.run.smithi092.stdout:(17/29): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2023-12-01T16:52:46.658 INFO:teuthology.orchestra.run.smithi092.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 624 kB/s | 41 kB 00:00 2023-12-01T16:52:46.683 INFO:teuthology.orchestra.run.smithi092.stdout:(19/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2023-12-01T16:52:46.810 INFO:teuthology.orchestra.run.smithi092.stdout:(20/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 195 kB/s | 24 kB 00:00 2023-12-01T16:52:46.900 INFO:teuthology.orchestra.run.smithi092.stdout:(21/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2023-12-01T16:52:46.926 INFO:teuthology.orchestra.run.smithi092.stdout:(22/29): liboath-2.6.7-2.el9.x86_64.rpm 415 kB/s | 49 kB 00:00 2023-12-01T16:52:46.959 INFO:teuthology.orchestra.run.smithi092.stdout:(23/29): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2023-12-01T16:52:47.026 INFO:teuthology.orchestra.run.smithi092.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2023-12-01T16:52:47.135 INFO:teuthology.orchestra.run.smithi092.stdout:(25/29): libarrow-9.0.0-9.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2023-12-01T16:52:47.177 INFO:teuthology.orchestra.run.smithi092.stdout:(26/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.3 MB/s | 837 kB 00:00 2023-12-01T16:52:47.236 INFO:teuthology.orchestra.run.smithi092.stdout:(27/29): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2023-12-01T16:52:47.419 INFO:teuthology.orchestra.run.smithi092.stdout:(28/29): librbd1-18.0.0-7570.gd710c09d.el9.x86_ 13 MB/s | 3.1 MB 00:00 2023-12-01T16:52:47.611 INFO:teuthology.orchestra.run.smithi092.stdout:(29/29): librados2-18.0.0-7570.gd710c09d.el9.x8 7.1 MB/s | 3.4 MB 00:00 2023-12-01T16:52:47.615 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:52:47.616 INFO:teuthology.orchestra.run.smithi092.stdout:Total 17 MB/s | 55 MB 00:03 2023-12-01T16:52:47.616 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:52:47.650 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:52:47.651 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:52:48.113 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:52:48.113 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:52:48.712 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:52:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:52:48.722 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:52:48.722 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 8.7 M 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 3.4 M 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 3.1 M 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 5.2 M 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 19 M 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 24 k 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-12-01T16:52:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 742 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 498 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 4.8 M 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 45 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 129 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 161 k 2023-12-01T16:52:48.724 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 320 k 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 298 k 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 99 k 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout:Install 27 Packages 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2023-12-01T16:52:48.725 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:52:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 55 M 2023-12-01T16:52:48.726 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:52:49.481 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:52:49.552 INFO:teuthology.orchestra.run.smithi067.stdout:(1/29): ceph-base-18.0.0-7570.gd710c09d.el9.x86 7.9 MB/s | 5.2 MB 00:00 2023-12-01T16:52:49.567 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-01T16:52:49.599 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-01T16:52:49.610 INFO:teuthology.orchestra.run.smithi067.stdout:(2/29): ceph-selinux-18.0.0-7570.gd710c09d.el9. 414 kB/s | 24 kB 00:00 2023-12-01T16:52:49.649 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-01T16:52:49.744 INFO:teuthology.orchestra.run.smithi067.stdout:(3/29): libcephfs2-18.0.0-7570.gd710c09d.el9.x8 5.4 MB/s | 742 kB 00:00 2023-12-01T16:52:49.844 INFO:teuthology.orchestra.run.smithi067.stdout:(4/29): ceph-radosgw-18.0.0-7570.gd710c09d.el9. 9.1 MB/s | 8.7 MB 00:00 2023-12-01T16:52:49.870 INFO:teuthology.orchestra.run.smithi067.stdout:(5/29): libradosstriper1-18.0.0-7570.gd710c09d. 3.9 MB/s | 498 kB 00:00 2023-12-01T16:52:49.920 INFO:teuthology.orchestra.run.smithi067.stdout:(6/29): python3-ceph-argparse-18.0.0-7570.gd710 907 kB/s | 45 kB 00:00 2023-12-01T16:52:49.966 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-01T16:52:49.978 INFO:teuthology.orchestra.run.smithi067.stdout:(7/29): python3-ceph-common-18.0.0-7570.gd710c0 2.2 MB/s | 129 kB 00:00 2023-12-01T16:52:50.023 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 5/31 2023-12-01T16:52:50.045 INFO:teuthology.orchestra.run.smithi067.stdout:(8/29): python3-cephfs-18.0.0-7570.gd710c09d.el 2.4 MB/s | 161 kB 00:00 2023-12-01T16:52:50.114 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 5/31 2023-12-01T16:52:50.137 INFO:teuthology.orchestra.run.smithi067.stdout:(9/29): librgw2-18.0.0-7570.gd710c09d.el9.x86_6 16 MB/s | 4.8 MB 00:00 2023-12-01T16:52:50.163 INFO:teuthology.orchestra.run.smithi067.stdout:(10/29): python3-rados-18.0.0-7570.gd710c09d.el 2.7 MB/s | 320 kB 00:00 2023-12-01T16:52:50.188 INFO:teuthology.orchestra.run.smithi067.stdout:(11/29): python3-rbd-18.0.0-7570.gd710c09d.el9. 5.8 MB/s | 298 kB 00:00 2023-12-01T16:52:50.201 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 6/31 2023-12-01T16:52:50.205 INFO:teuthology.orchestra.run.smithi067.stdout:(12/29): python3-rgw-18.0.0-7570.gd710c09d.el9. 2.3 MB/s | 99 kB 00:00 2023-12-01T16:52:50.233 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 7/31 2023-12-01T16:52:50.295 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 7/31 2023-12-01T16:52:50.324 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-01T16:52:50.330 INFO:teuthology.orchestra.run.smithi067.stdout:(13/29): boost-program-options-1.75.0-8.el9.x86 746 kB/s | 106 kB 00:00 2023-12-01T16:52:50.355 INFO:teuthology.orchestra.run.smithi067.stdout:(14/29): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2023-12-01T16:52:50.368 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:18.0.0-7570.gd710c09d.el9. 9/31 2023-12-01T16:52:50.372 INFO:teuthology.orchestra.run.smithi067.stdout:(15/29): libpmemobj-1.12.1-1.el9.x86_64.rpm 953 kB/s | 159 kB 00:00 2023-12-01T16:52:50.423 INFO:teuthology.orchestra.run.smithi067.stdout:(16/29): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2023-12-01T16:52:50.437 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 10/31 2023-12-01T16:52:50.448 INFO:teuthology.orchestra.run.smithi067.stdout:(17/29): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2023-12-01T16:52:50.460 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_6 11/31 2023-12-01T16:52:50.465 INFO:teuthology.orchestra.run.smithi067.stdout:(18/29): python3-prettytable-0.7.2-27.el9.noarc 997 kB/s | 41 kB 00:00 2023-12-01T16:52:50.526 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_6 11/31 2023-12-01T16:52:50.632 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-01T16:52:50.660 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-01T16:52:50.700 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-01T16:52:50.735 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-01T16:52:50.764 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-01T16:52:50.797 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-01T16:52:51.023 INFO:teuthology.orchestra.run.smithi067.stdout:(19/29): gperftools-libs-2.9.1-2.el9.x86_64.rpm 537 kB/s | 309 kB 00:00 2023-12-01T16:52:51.112 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-01T16:52:51.135 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 19/31 2023-12-01T16:52:51.216 INFO:teuthology.orchestra.run.smithi067.stdout:(20/29): ceph-common-18.0.0-7570.gd710c09d.el9. 8.2 MB/s | 19 MB 00:02 2023-12-01T16:52:51.226 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 19/31 2023-12-01T16:52:51.232 INFO:teuthology.orchestra.run.smithi067.stdout:(21/29): libarrow-doc-9.0.0-9.el9.noarch.rpm 117 kB/s | 24 kB 00:00 2023-12-01T16:52:51.257 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 20/31 2023-12-01T16:52:51.293 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-01T16:52:51.341 INFO:teuthology.orchestra.run.smithi067.stdout:(22/29): libunwind-1.6.2-1.el9.x86_64.rpm 620 kB/s | 67 kB 00:00 2023-12-01T16:52:51.399 INFO:teuthology.orchestra.run.smithi067.stdout:(23/29): libarrow-9.0.0-9.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2023-12-01T16:52:51.500 INFO:teuthology.orchestra.run.smithi067.stdout:(24/29): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2023-12-01T16:52:51.533 INFO:teuthology.orchestra.run.smithi067.stdout:(25/29): parquet-libs-9.0.0-9.el9.x86_64.rpm 4.3 MB/s | 837 kB 00:00 2023-12-01T16:52:51.550 INFO:teuthology.orchestra.run.smithi067.stdout:(26/29): liboath-2.6.7-2.el9.x86_64.rpm 145 kB/s | 49 kB 00:00 2023-12-01T16:52:51.715 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-01T16:52:51.747 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 23/31 2023-12-01T16:52:51.831 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 23/31 2023-12-01T16:52:51.834 INFO:teuthology.orchestra.run.smithi067.stdout:(27/29): librbd1-18.0.0-7570.gd710c09d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2023-12-01T16:52:51.901 INFO:teuthology.orchestra.run.smithi067.stdout:(28/29): librados2-18.0.0-7570.gd710c09d.el9.x8 9.1 MB/s | 3.4 MB 00:00 2023-12-01T16:52:51.902 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 24/31 2023-12-01T16:52:51.932 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:18.0.0-7570.gd710c09d.el9.x8 25/31 2023-12-01T16:52:52.226 INFO:teuthology.orchestra.run.smithi067.stdout:(29/29): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2023-12-01T16:52:52.230 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:52:52.231 INFO:teuthology.orchestra.run.smithi067.stdout:Total 16 MB/s | 55 MB 00:03 2023-12-01T16:52:52.231 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:52:52.272 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:52:52.273 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:52:52.750 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:52:52.867 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:52:54.095 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 26/31 2023-12-01T16:52:54.131 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:52:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 26/31 2023-12-01T16:52:54.256 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/31 2023-12-01T16:52:54.286 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/31 2023-12-01T16:52:54.345 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/31 2023-12-01T16:52:54.695 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 26/31 2023-12-01T16:52:54.732 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 27/31 2023-12-01T16:52:54.741 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/31 2023-12-01T16:52:54.768 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 5/31 2023-12-01T16:52:54.811 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 27/31 2023-12-01T16:52:54.811 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-01T16:52:54.811 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-01T16:52:54.811 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:52:54.842 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 28/31 2023-12-01T16:52:54.860 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 5/31 2023-12-01T16:52:54.950 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 6/31 2023-12-01T16:52:54.978 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 7/31 2023-12-01T16:52:55.025 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 7/31 2023-12-01T16:52:55.053 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/31 2023-12-01T16:52:55.097 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:18.0.0-7570.gd710c09d.el9. 9/31 2023-12-01T16:52:55.167 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 10/31 2023-12-01T16:52:55.189 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_6 11/31 2023-12-01T16:52:55.238 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_6 11/31 2023-12-01T16:52:55.348 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/31 2023-12-01T16:52:55.385 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 13/31 2023-12-01T16:52:55.429 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/31 2023-12-01T16:52:55.461 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/31 2023-12-01T16:52:55.490 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 16/31 2023-12-01T16:52:55.523 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 17/31 2023-12-01T16:52:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 18/31 2023-12-01T16:52:55.878 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 19/31 2023-12-01T16:52:55.952 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 19/31 2023-12-01T16:52:55.982 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 20/31 2023-12-01T16:52:56.018 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 21/31 2023-12-01T16:52:56.479 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 22/31 2023-12-01T16:52:56.505 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 23/31 2023-12-01T16:52:56.579 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 23/31 2023-12-01T16:52:56.657 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 24/31 2023-12-01T16:52:56.701 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:18.0.0-7570.gd710c09d.el9.x8 25/31 2023-12-01T16:52:58.950 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 26/31 2023-12-01T16:52:58.989 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 26/31 2023-12-01T16:52:59.577 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 26/31 2023-12-01T16:52:59.625 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 27/31 2023-12-01T16:52:59.714 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 27/31 2023-12-01T16:52:59.714 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-12-01T16:52:59.714 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-12-01T16:52:59.714 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:52:59.745 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 28/31 2023-12-01T16:53:04.422 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 28/31 2023-12-01T16:53:04.422 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2023-12-01T16:53:04.422 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2023-12-01T16:53:04.422 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2023-12-01T16:53:04.423 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2023-12-01T16:53:04.423 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2023-12-01T16:53:04.423 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2023-12-01T16:53:04.423 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2023-12-01T16:53:04.423 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:05.252 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 29/31 2023-12-01T16:53:05.277 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 29/31 2023-12-01T16:53:05.277 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:05.277 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-01T16:53:05.277 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-01T16:53:05.277 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-01T16:53:05.277 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:05.279 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-01T16:53:05.354 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-01T16:53:05.355 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-01T16:53:06.427 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-01T16:53:06.427 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 1/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 2/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 3/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 4/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 5/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_6 6/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 7/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7570.gd710c09d.el9. 8/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:18.0.0-7570.gd710c09d.el9.x8 9/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 10/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 11/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 12/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 13/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-01T16:53:06.428 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 28/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-01T16:53:06.429 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 30/31 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-01T16:53:07.417 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.418 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-01T16:53:07.419 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:07.420 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:07.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-test 2023-12-01T16:53:08.389 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:27 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:08.546 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 45 M 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-01T16:53:08.547 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 163 k 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:08.548 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 46 M 2023-12-01T16:53:08.549 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 199 M 2023-12-01T16:53:08.549 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:53:08.716 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-01T16:53:08.766 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2023-12-01T16:53:08.808 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-01T16:53:08.842 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-01T16:53:08.942 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): libcephsqlite-18.0.0-7570.gd710c09d.el9. 416 kB/s | 163 kB 00:00 2023-12-01T16:53:09.815 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 28/31 2023-12-01T16:53:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2023-12-01T16:53:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2023-12-01T16:53:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2023-12-01T16:53:09.815 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2023-12-01T16:53:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2023-12-01T16:53:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2023-12-01T16:53:09.816 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2023-12-01T16:53:09.816 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:10.652 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 29/31 2023-12-01T16:53:10.677 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 29/31 2023-12-01T16:53:10.678 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:10.678 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-12-01T16:53:10.678 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-01T16:53:10.678 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-12-01T16:53:10.678 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:10.679 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-01T16:53:10.746 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 30/31 2023-12-01T16:53:10.746 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-01T16:53:10.775 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): ceph-test-18.0.0-7570.gd710c09d.el9.x86_ 20 MB/s | 45 MB 00:02 2023-12-01T16:53:10.778 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:10.779 INFO:teuthology.orchestra.run.smithi092.stdout:Total 21 MB/s | 46 MB 00:02 2023-12-01T16:53:10.779 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:53:10.798 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:53:10.798 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:53:11.176 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:53:11.176 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:53:11.758 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:53:11.835 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-01T16:53:11.872 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-01T16:53:11.900 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 31/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 1/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 2/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 3/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 4/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 5/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_6 6/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 7/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7570.gd710c09d.el9. 8/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:18.0.0-7570.gd710c09d.el9.x8 9/31 2023-12-01T16:53:11.912 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 10/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 11/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 12/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 13/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 20/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 21/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 22/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 23/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 24/31 2023-12-01T16:53:11.913 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 25/31 2023-12-01T16:53:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 26/31 2023-12-01T16:53:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 27/31 2023-12-01T16:53:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 28/31 2023-12-01T16:53:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 29/31 2023-12-01T16:53:11.914 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 30/31 2023-12-01T16:53:11.943 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-01T16:53:11.962 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 5/6 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 31/31 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-12-01T16:53:12.753 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-9.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-12-01T16:53:12.754 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:12.755 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2023-12-01T16:53:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2023-12-01T16:53:12.756 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:12.756 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:13.075 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test 2023-12-01T16:53:13.681 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:28 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:13.835 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:13.836 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:53:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:53:13.836 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 45 M 2023-12-01T16:53:13.836 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.6-16.el9 CentOS-AppStream 186 k 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 163 k 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2023-12-01T16:53:13.837 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:13.838 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 46 M 2023-12-01T16:53:13.838 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 199 M 2023-12-01T16:53:13.838 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:53:14.001 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 186 kB 00:00 2023-12-01T16:53:14.042 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2023-12-01T16:53:14.084 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2023-12-01T16:53:14.118 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2023-12-01T16:53:14.151 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): libcephsqlite-18.0.0-7570.gd710c09d.el9. 521 kB/s | 163 kB 00:00 2023-12-01T16:53:16.077 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): ceph-test-18.0.0-7570.gd710c09d.el9.x86_ 20 MB/s | 45 MB 00:02 2023-12-01T16:53:16.079 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:16.080 INFO:teuthology.orchestra.run.smithi067.stdout:Total 21 MB/s | 46 MB 00:02 2023-12-01T16:53:16.080 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:53:16.103 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:53:16.103 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:53:16.271 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 5/6 2023-12-01T16:53:16.352 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 6/6 2023-12-01T16:53:16.493 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:53:16.493 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:53:17.087 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:53:17.145 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2023-12-01T16:53:17.201 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2023-12-01T16:53:17.234 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2023-12-01T16:53:17.278 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2023-12-01T16:53:17.304 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 5/6 2023-12-01T16:53:18.794 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 6/6 2023-12-01T16:53:18.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 1/6 2023-12-01T16:53:18.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 2/6 2023-12-01T16:53:18.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-01T16:53:18.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-01T16:53:18.794 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-01T16:53:19.206 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-01T16:53:19.207 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:19.207 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:53:19.207 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:19.207 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-16.el9.x86_64 2023-12-01T16:53:19.207 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:19.207 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-01T16:53:19.207 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-01T16:53:19.208 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-01T16:53:19.208 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:19.208 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:19.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph 2023-12-01T16:53:20.050 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:39 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:20.206 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:20.208 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 6.6 k 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 2.1 M 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.5 M 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 243 k 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 4.4 M 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 18 M 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-01T16:53:20.209 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-01T16:53:20.210 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:53:20.211 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2023-12-01T16:53:20.212 INFO:teuthology.orchestra.run.smithi092.stdout:Install 36 Packages 2023-12-01T16:53:20.212 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:20.212 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 30 M 2023-12-01T16:53:20.212 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 112 M 2023-12-01T16:53:20.212 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:53:20.685 INFO:teuthology.orchestra.run.smithi092.stdout:(1/36): ceph-18.0.0-7570.gd710c09d.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2023-12-01T16:53:20.927 INFO:teuthology.orchestra.run.smithi092.stdout:(2/36): ceph-mds-18.0.0-7570.gd710c09d.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2023-12-01T16:53:21.236 INFO:teuthology.orchestra.run.smithi092.stdout:(3/36): ceph-mon-18.0.0-7570.gd710c09d.el9.x86_ 8.1 MB/s | 4.4 MB 00:00 2023-12-01T16:53:21.336 INFO:teuthology.orchestra.run.smithi092.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7570.gd710 2.4 MB/s | 243 kB 00:00 2023-12-01T16:53:21.453 INFO:teuthology.orchestra.run.smithi092.stdout:(5/36): ceph-mgr-18.0.0-7570.gd710c09d.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2023-12-01T16:53:21.478 INFO:teuthology.orchestra.run.smithi092.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2023-12-01T16:53:21.504 INFO:teuthology.orchestra.run.smithi092.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2023-12-01T16:53:21.520 INFO:teuthology.orchestra.run.smithi092.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 755 kB/s | 32 kB 00:00 2023-12-01T16:53:21.537 INFO:teuthology.orchestra.run.smithi092.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-01T16:53:21.554 INFO:teuthology.orchestra.run.smithi092.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2023-12-01T16:53:21.579 INFO:teuthology.orchestra.run.smithi092.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2023-12-01T16:53:21.630 INFO:teuthology.orchestra.run.smithi092.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-01T16:53:21.655 INFO:teuthology.orchestra.run.smithi092.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2023-12-01T16:53:21.688 INFO:teuthology.orchestra.run.smithi092.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2023-12-01T16:53:21.713 INFO:teuthology.orchestra.run.smithi092.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 1.9 MB/s | 113 kB 00:00 2023-12-01T16:53:21.738 INFO:teuthology.orchestra.run.smithi092.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2023-12-01T16:53:21.875 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 5/6 2023-12-01T16:53:21.914 INFO:teuthology.orchestra.run.smithi092.stdout:(17/36): ceph-osd-18.0.0-7570.gd710c09d.el9.x86 18 MB/s | 18 MB 00:00 2023-12-01T16:53:21.956 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 6/6 2023-12-01T16:53:21.964 INFO:teuthology.orchestra.run.smithi092.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 89 kB/s | 22 kB 00:00 2023-12-01T16:53:22.081 INFO:teuthology.orchestra.run.smithi092.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 441 kB/s | 151 kB 00:00 2023-12-01T16:53:22.106 INFO:teuthology.orchestra.run.smithi092.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 226 kB/s | 43 kB 00:00 2023-12-01T16:53:22.131 INFO:teuthology.orchestra.run.smithi092.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2023-12-01T16:53:22.748 INFO:teuthology.orchestra.run.smithi092.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 17 kB/s | 11 kB 00:00 2023-12-01T16:53:22.774 INFO:teuthology.orchestra.run.smithi092.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 27 kB/s | 18 kB 00:00 2023-12-01T16:53:22.799 INFO:teuthology.orchestra.run.smithi092.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 500 kB/s | 358 kB 00:00 2023-12-01T16:53:22.824 INFO:teuthology.orchestra.run.smithi092.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2023-12-01T16:53:22.874 INFO:teuthology.orchestra.run.smithi092.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2023-12-01T16:53:22.908 INFO:teuthology.orchestra.run.smithi092.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 181 kB/s | 20 kB 00:00 2023-12-01T16:53:22.950 INFO:teuthology.orchestra.run.smithi092.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2023-12-01T16:53:22.975 INFO:teuthology.orchestra.run.smithi092.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 307 kB/s | 46 kB 00:00 2023-12-01T16:53:23.025 INFO:teuthology.orchestra.run.smithi092.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2023-12-01T16:53:23.084 INFO:teuthology.orchestra.run.smithi092.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 614 kB/s | 36 kB 00:00 2023-12-01T16:53:23.159 INFO:teuthology.orchestra.run.smithi092.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2023-12-01T16:53:23.184 INFO:teuthology.orchestra.run.smithi092.stdout:(33/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 430 kB/s | 90 kB 00:00 2023-12-01T16:53:23.210 INFO:teuthology.orchestra.run.smithi092.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 903 kB/s | 272 kB 00:00 2023-12-01T16:53:23.243 INFO:teuthology.orchestra.run.smithi092.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2023-12-01T16:53:23.527 INFO:teuthology.orchestra.run.smithi092.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 58 kB/s | 20 kB 00:00 2023-12-01T16:53:23.530 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:23.530 INFO:teuthology.orchestra.run.smithi092.stdout:Total 9.1 MB/s | 30 MB 00:03 2023-12-01T16:53:23.543 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:53:23.586 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:53:23.587 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:53:24.007 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:53:24.008 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:53:24.321 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 6/6 2023-12-01T16:53:24.321 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 1/6 2023-12-01T16:53:24.321 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 2/6 2023-12-01T16:53:24.321 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.6-16.el9.x86_64 3/6 2023-12-01T16:53:24.321 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 4/6 2023-12-01T16:53:24.321 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 5/6 2023-12-01T16:53:24.657 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 6/6 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.6-16.el9.x86_64 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:24.658 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:24.928 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph 2023-12-01T16:53:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:53:25.405 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-01T16:53:25.436 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-01T16:53:25.465 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-01T16:53:25.529 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:40 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:25.533 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-01T16:53:25.572 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-01T16:53:25.605 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-01T16:53:25.683 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:25.686 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2023-12-01T16:53:25.686 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:53:25.686 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2023-12-01T16:53:25.686 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:53:25.686 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 6.6 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 2.1 M 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.5 M 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 243 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 4.4 M 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 18 M 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2023-12-01T16:53:25.687 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-12-01T16:53:25.688 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-4.el9 CentOS-BaseOS 187 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout:========================================================================================= 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout:Install 36 Packages 2023-12-01T16:53:25.689 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:25.690 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 30 M 2023-12-01T16:53:25.690 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 112 M 2023-12-01T16:53:25.690 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:53:25.694 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-01T16:53:25.768 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-01T16:53:25.807 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-01T16:53:25.880 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-01T16:53:25.933 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-01T16:53:25.971 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-01T16:53:26.019 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-01T16:53:26.075 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-01T16:53:26.195 INFO:teuthology.orchestra.run.smithi067.stdout:(1/36): ceph-18.0.0-7570.gd710c09d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2023-12-01T16:53:26.268 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-01T16:53:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-01T16:53:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-01T16:53:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-01T16:53:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-01T16:53:26.488 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-01T16:53:26.530 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-01T16:53:26.541 INFO:teuthology.orchestra.run.smithi067.stdout:(2/36): ceph-mgr-18.0.0-7570.gd710c09d.el9.x86_ 2.9 MB/s | 1.5 MB 00:00 2023-12-01T16:53:26.563 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-01T16:53:26.604 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-01T16:53:26.612 INFO:teuthology.orchestra.run.smithi067.stdout:(3/36): ceph-mon-18.0.0-7570.gd710c09d.el9.x86_ 11 MB/s | 4.4 MB 00:00 2023-12-01T16:53:26.639 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-01T16:53:26.670 INFO:teuthology.orchestra.run.smithi067.stdout:(4/36): ceph-mgr-modules-core-18.0.0-7570.gd710 4.1 MB/s | 243 kB 00:00 2023-12-01T16:53:26.682 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-01T16:53:26.772 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-01T16:53:26.787 INFO:teuthology.orchestra.run.smithi067.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2023-12-01T16:53:26.813 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-01T16:53:26.821 INFO:teuthology.orchestra.run.smithi067.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2023-12-01T16:53:26.855 INFO:teuthology.orchestra.run.smithi067.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2023-12-01T16:53:26.886 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-01T16:53:26.888 INFO:teuthology.orchestra.run.smithi067.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2023-12-01T16:53:26.930 INFO:teuthology.orchestra.run.smithi067.stdout:(9/36): ceph-mds-18.0.0-7570.gd710c09d.el9.x86_ 2.4 MB/s | 2.1 MB 00:00 2023-12-01T16:53:26.955 INFO:teuthology.orchestra.run.smithi067.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 576 kB/s | 39 kB 00:00 2023-12-01T16:53:26.981 INFO:teuthology.orchestra.run.smithi067.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2023-12-01T16:53:27.021 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-01T16:53:27.031 INFO:teuthology.orchestra.run.smithi067.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2023-12-01T16:53:27.057 INFO:teuthology.orchestra.run.smithi067.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2023-12-01T16:53:27.082 INFO:teuthology.orchestra.run.smithi067.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2023-12-01T16:53:27.107 INFO:teuthology.orchestra.run.smithi067.stdout:(15/36): python3-requests-2.25.1-7.el9.noarch.r 2.2 MB/s | 113 kB 00:00 2023-12-01T16:53:27.124 INFO:teuthology.orchestra.run.smithi067.stdout:(16/36): python3-urllib3-1.26.5-4.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2023-12-01T16:53:27.148 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-01T16:53:27.312 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7570.gd710c09d.el9. 31/36 2023-12-01T16:53:27.342 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 32/36 2023-12-01T16:53:27.367 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 32/36 2023-12-01T16:53:27.367 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:27.367 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-01T16:53:27.368 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-01T16:53:27.368 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-01T16:53:27.368 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:27.466 INFO:teuthology.orchestra.run.smithi067.stdout:(17/36): ceph-osd-18.0.0-7570.gd710c09d.el9.x86 19 MB/s | 18 MB 00:00 2023-12-01T16:53:27.482 INFO:teuthology.orchestra.run.smithi067.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 59 kB/s | 22 kB 00:00 2023-12-01T16:53:27.616 INFO:teuthology.orchestra.run.smithi067.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 307 kB/s | 151 kB 00:00 2023-12-01T16:53:28.891 INFO:teuthology.orchestra.run.smithi067.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 30 kB/s | 43 kB 00:01 2023-12-01T16:53:28.936 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 33/36 2023-12-01T16:53:28.969 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 33/36 2023-12-01T16:53:28.969 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:28.969 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-01T16:53:28.969 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-01T16:53:28.969 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-01T16:53:28.969 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:29.058 INFO:teuthology.orchestra.run.smithi067.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 109 kB/s | 172 kB 00:01 2023-12-01T16:53:29.209 INFO:teuthology.orchestra.run.smithi067.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 118 kB/s | 18 kB 00:00 2023-12-01T16:53:29.225 INFO:teuthology.orchestra.run.smithi067.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 32 kB/s | 11 kB 00:00 2023-12-01T16:53:29.284 INFO:teuthology.orchestra.run.smithi067.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 215 kB/s | 358 kB 00:01 2023-12-01T16:53:29.301 INFO:teuthology.orchestra.run.smithi067.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2023-12-01T16:53:29.318 INFO:teuthology.orchestra.run.smithi067.stdout:(26/36): python3-jaraco-collections-3.0.0-8.el9 212 kB/s | 23 kB 00:00 2023-12-01T16:53:29.420 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 34/36 2023-12-01T16:53:29.444 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 34/36 2023-12-01T16:53:29.444 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:29.444 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-01T16:53:29.444 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-01T16:53:29.444 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-01T16:53:29.444 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 35/36 2023-12-01T16:53:29.668 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 35/36 2023-12-01T16:53:29.668 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:29.669 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-01T16:53:29.669 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-01T16:53:29.669 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-01T16:53:29.669 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:29.734 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 36/36 2023-12-01T16:53:29.743 INFO:teuthology.orchestra.run.smithi067.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 185 kB/s | 79 kB 00:00 2023-12-01T16:53:29.760 INFO:teuthology.orchestra.run.smithi067.stdout:(28/36): python3-jaraco-text-3.2.0-6.el9.noarch 41 kB/s | 20 kB 00:00 2023-12-01T16:53:29.777 INFO:teuthology.orchestra.run.smithi067.stdout:(29/36): python3-logutils-0.3.5-21.el9.noarch.r 97 kB/s | 46 kB 00:00 2023-12-01T16:53:29.836 INFO:teuthology.orchestra.run.smithi067.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 218 kB/s | 16 kB 00:00 2023-12-01T16:53:29.877 INFO:teuthology.orchestra.run.smithi067.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 896 kB/s | 90 kB 00:00 2023-12-01T16:53:29.944 INFO:teuthology.orchestra.run.smithi067.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2023-12-01T16:53:29.995 INFO:teuthology.orchestra.run.smithi067.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 225 kB/s | 36 kB 00:00 2023-12-01T16:53:30.028 INFO:teuthology.orchestra.run.smithi067.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2023-12-01T16:53:30.177 INFO:teuthology.orchestra.run.smithi067.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2023-12-01T16:53:30.262 INFO:teuthology.orchestra.run.smithi067.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 75 kB/s | 20 kB 00:00 2023-12-01T16:53:30.265 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:30.265 INFO:teuthology.orchestra.run.smithi067.stdout:Total 6.6 MB/s | 30 MB 00:04 2023-12-01T16:53:30.279 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:53:30.325 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:53:30.325 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:53:30.759 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:53:30.759 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:53:31.544 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 36/36 2023-12-01T16:53:31.544 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 1/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 2/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 3/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 4/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 5/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7570.gd710c09d.el9. 6/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-01T16:53:31.545 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-01T16:53:31.546 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-01T16:53:31.547 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-01T16:53:31.547 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-01T16:53:31.547 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-01T16:53:31.547 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-01T16:53:32.145 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:53:32.197 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2023-12-01T16:53:32.237 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2023-12-01T16:53:32.274 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2023-12-01T16:53:32.338 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2023-12-01T16:53:32.381 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2023-12-01T16:53:32.414 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2023-12-01T16:53:32.508 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2023-12-01T16:53:32.594 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2023-12-01T16:53:32.642 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:32.695 INFO:teuthology.orchestra.run.smithi092.stdout: lua-5.4.4-4.el9.x86_64 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-01T16:53:32.696 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-01T16:53:32.697 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-01T16:53:32.698 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-01T16:53:32.698 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:32.698 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:32.715 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2023-12-01T16:53:32.768 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-4.el9.noarch 11/36 2023-12-01T16:53:32.806 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 12/36 2023-12-01T16:53:32.856 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2023-12-01T16:53:32.911 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2023-12-01T16:53:33.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base 2023-12-01T16:53:33.109 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2023-12-01T16:53:33.160 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2023-12-01T16:53:33.198 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2023-12-01T16:53:33.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2023-12-01T16:53:33.263 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2023-12-01T16:53:33.341 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2023-12-01T16:53:33.381 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-01T16:53:33.415 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2023-12-01T16:53:33.455 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2023-12-01T16:53:33.490 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2023-12-01T16:53:33.533 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2023-12-01T16:53:33.603 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:52 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:33.624 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2023-12-01T16:53:33.664 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2023-12-01T16:53:33.698 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:53:33.738 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2023-12-01T16:53:33.749 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:33.750 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:53:33.750 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:33.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install cephadm 2023-12-01T16:53:33.877 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2023-12-01T16:53:34.000 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-01T16:53:34.171 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7570.gd710c09d.el9. 31/36 2023-12-01T16:53:34.201 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 32/36 2023-12-01T16:53:34.227 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 32/36 2023-12-01T16:53:34.227 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:34.227 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-12-01T16:53:34.227 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-01T16:53:34.228 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-12-01T16:53:34.228 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:34.382 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:53 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:34.526 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:34.526 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:34.526 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 775 k 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 775 k 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 788 k 2023-12-01T16:53:34.527 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:53:35.002 INFO:teuthology.orchestra.run.smithi092.stdout:cephadm-18.0.0-7570.gd710c09d.el9.noarch.rpm 1.6 MB/s | 775 kB 00:00 2023-12-01T16:53:35.002 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:35.002 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.6 MB/s | 775 kB 00:00 2023-12-01T16:53:35.002 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:53:35.007 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:53:35.008 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:53:35.014 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:53:35.014 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:53:35.085 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:53:35.534 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:35.622 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:35.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 33/36 2023-12-01T16:53:35.865 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 33/36 2023-12-01T16:53:35.865 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:35.865 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-12-01T16:53:35.865 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-01T16:53:35.865 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-12-01T16:53:35.865 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:35.994 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:36.278 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:36.278 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:36.278 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:53:36.278 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:36.278 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:36.278 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:36.329 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 34/36 2023-12-01T16:53:36.355 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 34/36 2023-12-01T16:53:36.355 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:36.355 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-12-01T16:53:36.355 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-01T16:53:36.355 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-12-01T16:53:36.355 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:36.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-immutable-object-cache 2023-12-01T16:53:36.571 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 35/36 2023-12-01T16:53:36.596 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 35/36 2023-12-01T16:53:36.596 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:36.596 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-12-01T16:53:36.596 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-01T16:53:36.596 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-12-01T16:53:36.596 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:36.663 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 36/36 2023-12-01T16:53:37.060 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:56 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:37.210 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 145 k 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:37.211 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:53:37.212 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:37.212 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 145 k 2023-12-01T16:53:37.212 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 435 k 2023-12-01T16:53:37.212 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:53:37.574 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-immutable-object-cache-18.0.0-7570.gd710c0 401 kB/s | 145 kB 00:00 2023-12-01T16:53:37.574 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:37.575 INFO:teuthology.orchestra.run.smithi092.stdout:Total 399 kB/s | 145 kB 00:00 2023-12-01T16:53:37.575 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:53:37.582 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:53:37.582 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:53:37.634 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:53:37.634 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:53:37.782 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:53:37.841 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d. 1/1 2023-12-01T16:53:37.865 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d. 1/1 2023-12-01T16:53:37.865 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:37.865 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-01T16:53:37.865 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:38.398 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 36/36 2023-12-01T16:53:38.398 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 1/36 2023-12-01T16:53:38.398 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 2/36 2023-12-01T16:53:38.398 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 3/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 4/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 5/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7570.gd710c09d.el9. 6/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 16/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-4.el9.noarch 17/36 2023-12-01T16:53:38.399 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2023-12-01T16:53:38.400 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2023-12-01T16:53:38.401 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2023-12-01T16:53:38.836 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d. 1/1 2023-12-01T16:53:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:38.837 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:53:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:38.837 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:38.837 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:39.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:39.242 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-12-01T16:53:39.243 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-7.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-4.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-12-01T16:53:39.244 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:39.245 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:39.601 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base 2023-12-01T16:53:39.635 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:58 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:39.730 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:53:39.779 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:39.780 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:53:39.780 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:39.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-dashboard 2023-12-01T16:53:40.208 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:55 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:40.308 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:53:40.362 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:40.363 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:53:40.363 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:40.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install cephadm 2023-12-01T16:53:40.420 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:59 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:40.564 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:40.565 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:40.565 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:53:40.565 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:40.565 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 1.8 M 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 28 k 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 15 k 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout:Install 7 Packages 2023-12-01T16:53:40.566 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:40.567 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 2.1 M 2023-12-01T16:53:40.567 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 8.7 M 2023-12-01T16:53:40.567 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:53:40.927 INFO:teuthology.orchestra.run.smithi092.stdout:(1/7): ceph-grafana-dashboards-18.0.0-7570.gd71 150 kB/s | 28 kB 00:00 2023-12-01T16:53:41.017 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:56 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:41.019 INFO:teuthology.orchestra.run.smithi092.stdout:(2/7): ceph-prometheus-alerts-18.0.0-7570.gd710 54 kB/s | 15 kB 00:00 2023-12-01T16:53:41.169 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 775 k 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 775 k 2023-12-01T16:53:41.170 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 788 k 2023-12-01T16:53:41.171 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:53:41.186 INFO:teuthology.orchestra.run.smithi092.stdout:(3/7): ceph-mgr-dashboard-18.0.0-7570.gd710c09d 4.0 MB/s | 1.8 MB 00:00 2023-12-01T16:53:41.628 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm-18.0.0-7570.gd710c09d.el9.noarch.rpm 1.7 MB/s | 775 kB 00:00 2023-12-01T16:53:41.629 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:41.629 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.6 MB/s | 775 kB 00:00 2023-12-01T16:53:41.629 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:53:41.634 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:53:41.634 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:53:41.640 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:53:41.641 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:53:41.719 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:53:42.153 INFO:teuthology.orchestra.run.smithi092.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 7.4 kB/s | 9.0 kB 00:01 2023-12-01T16:53:42.162 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:42.170 INFO:teuthology.orchestra.run.smithi092.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 31 kB/s | 31 kB 00:00 2023-12-01T16:53:42.186 INFO:teuthology.orchestra.run.smithi092.stdout:(6/7): python3-jwt-2.4.0-1.el9.noarch.rpm 35 kB/s | 41 kB 00:01 2023-12-01T16:53:42.242 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:42.596 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:42.870 INFO:teuthology.orchestra.run.smithi092.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 263 kB/s | 188 kB 00:00 2023-12-01T16:53:42.872 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:42.872 INFO:teuthology.orchestra.run.smithi092.stdout:Total 916 kB/s | 2.1 MB 00:02 2023-12-01T16:53:42.873 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:53:42.881 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:53:42.881 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:53:42.897 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:53:42.897 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:42.898 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:53:42.898 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:42.898 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:42.898 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:42.954 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:53:42.954 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:53:43.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache 2023-12-01T16:53:43.255 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:53:43.330 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-01T16:53:43.363 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-01T16:53:43.409 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-01T16:53:43.442 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-01T16:53:43.472 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7570.gd710c09d.el9.n 5/7 2023-12-01T16:53:43.701 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:58 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:43.708 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7570.gd710c09d.el9. 6/7 2023-12-01T16:53:43.740 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarc 7/7 2023-12-01T16:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:53:43.852 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 145 k 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 145 k 2023-12-01T16:53:43.853 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 435 k 2023-12-01T16:53:43.854 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:53:44.216 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-18.0.0-7570.gd710c0 400 kB/s | 145 kB 00:00 2023-12-01T16:53:44.217 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:44.217 INFO:teuthology.orchestra.run.smithi067.stdout:Total 398 kB/s | 145 kB 00:00 2023-12-01T16:53:44.217 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:53:44.225 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:53:44.226 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:53:44.280 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:53:44.280 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:53:44.327 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarc 7/7 2023-12-01T16:53:44.328 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7570.gd710c09d.el9. 1/7 2023-12-01T16:53:44.328 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarc 2/7 2023-12-01T16:53:44.328 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7570.gd710c09d.el9.n 3/7 2023-12-01T16:53:44.328 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-01T16:53:44.328 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-01T16:53:44.328 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-01T16:53:44.438 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:53:44.484 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d. 1/1 2023-12-01T16:53:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d. 1/1 2023-12-01T16:53:44.510 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:53:44.510 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-12-01T16:53:44.510 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:44.634 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:44.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-01T16:53:45.442 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d. 1/1 2023-12-01T16:53:45.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:45.443 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:53:45.443 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:53:45.443 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:45.443 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:45.449 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:04 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:45.593 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 7.4 M 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-01T16:53:45.594 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout:Install 12 Packages 2023-12-01T16:53:45.595 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:45.596 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 37 M 2023-12-01T16:53:45.596 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 209 M 2023-12-01T16:53:45.596 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:53:45.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr 2023-12-01T16:53:46.001 INFO:teuthology.orchestra.run.smithi092.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 235 kB/s | 32 kB 00:00 2023-12-01T16:53:46.043 INFO:teuthology.orchestra.run.smithi092.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 438 kB/s | 17 kB 00:00 2023-12-01T16:53:46.102 INFO:teuthology.orchestra.run.smithi092.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2023-12-01T16:53:46.127 INFO:teuthology.orchestra.run.smithi092.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 417 kB/s | 34 kB 00:00 2023-12-01T16:53:46.267 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:01 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:46.368 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-mgr-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:53:46.419 INFO:teuthology.orchestra.run.smithi092.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-12-01T16:53:46.421 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:46.422 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:53:46.422 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:46.482 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-dashboard 2023-12-01T16:53:46.494 INFO:teuthology.orchestra.run.smithi092.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 13 MB/s | 4.8 MB 00:00 2023-12-01T16:53:46.519 INFO:teuthology.orchestra.run.smithi092.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2023-12-01T16:53:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 4.8 MB/s | 803 kB 00:00 2023-12-01T16:53:46.837 INFO:teuthology.orchestra.run.smithi092.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2023-12-01T16:53:46.862 INFO:teuthology.orchestra.run.smithi092.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2023-12-01T16:53:47.004 INFO:teuthology.orchestra.run.smithi092.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2023-12-01T16:53:47.089 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:02 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:47.240 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:47.241 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:47.241 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:53:47.241 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 1.8 M 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 28 k 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 15 k 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:53:47.242 INFO:teuthology.orchestra.run.smithi067.stdout:Install 7 Packages 2023-12-01T16:53:47.243 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:47.243 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.1 M 2023-12-01T16:53:47.243 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 8.7 M 2023-12-01T16:53:47.243 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:53:47.569 INFO:teuthology.orchestra.run.smithi067.stdout:(1/7): ceph-prometheus-alerts-18.0.0-7570.gd710 76 kB/s | 15 kB 00:00 2023-12-01T16:53:47.595 INFO:teuthology.orchestra.run.smithi067.stdout:(2/7): ceph-grafana-dashboards-18.0.0-7570.gd71 124 kB/s | 28 kB 00:00 2023-12-01T16:53:47.645 INFO:teuthology.orchestra.run.smithi067.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 120 kB/s | 9.0 kB 00:00 2023-12-01T16:53:47.678 INFO:teuthology.orchestra.run.smithi067.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 490 kB/s | 41 kB 00:00 2023-12-01T16:53:47.696 INFO:teuthology.orchestra.run.smithi067.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 611 kB/s | 31 kB 00:00 2023-12-01T16:53:47.746 INFO:teuthology.orchestra.run.smithi067.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2023-12-01T16:53:48.129 INFO:teuthology.orchestra.run.smithi067.stdout:(7/7): ceph-mgr-dashboard-18.0.0-7570.gd710c09d 2.3 MB/s | 1.8 MB 00:00 2023-12-01T16:53:48.131 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:48.131 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.3 MB/s | 2.1 MB 00:00 2023-12-01T16:53:48.131 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:53:48.140 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:53:48.140 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:53:48.214 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:53:48.214 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:53:48.270 INFO:teuthology.orchestra.run.smithi092.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 3.1 MB/s | 7.4 MB 00:02 2023-12-01T16:53:48.273 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:48.273 INFO:teuthology.orchestra.run.smithi092.stdout:Total 14 MB/s | 37 MB 00:02 2023-12-01T16:53:48.287 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:53:48.313 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:53:48.313 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:53:48.525 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:53:48.581 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2023-12-01T16:53:48.615 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2023-12-01T16:53:48.663 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2023-12-01T16:53:48.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2023-12-01T16:53:48.701 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:53:48.702 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:53:48.723 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7570.gd710c09d.el9.n 5/7 2023-12-01T16:53:48.969 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7570.gd710c09d.el9. 6/7 2023-12-01T16:53:48.999 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarc 7/7 2023-12-01T16:53:49.391 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:53:49.462 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-01T16:53:49.491 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-01T16:53:49.546 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarc 7/7 2023-12-01T16:53:49.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7570.gd710c09d.el9. 1/7 2023-12-01T16:53:49.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarc 2/7 2023-12-01T16:53:49.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7570.gd710c09d.el9.n 3/7 2023-12-01T16:53:49.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2023-12-01T16:53:49.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2023-12-01T16:53:49.546 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2023-12-01T16:53:49.582 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-01T16:53:49.619 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-01T16:53:49.801 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-01T16:53:49.835 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-01T16:53:49.907 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2023-12-01T16:53:49.908 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:49.909 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:53:49.920 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-01T16:53:50.107 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-01T16:53:50.573 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-01T16:53:50.696 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-01T16:53:50.709 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:05 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:53:50.862 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:53:50.863 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 7.4 M 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2023-12-01T16:53:50.864 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-12-01T16:53:50.865 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-12-01T16:53:50.865 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-12-01T16:53:50.865 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:50.865 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:53:50.865 INFO:teuthology.orchestra.run.smithi067.stdout:============================================================================================ 2023-12-01T16:53:50.865 INFO:teuthology.orchestra.run.smithi067.stdout:Install 12 Packages 2023-12-01T16:53:50.865 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:50.866 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 37 M 2023-12-01T16:53:50.866 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 209 M 2023-12-01T16:53:50.866 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:53:51.086 INFO:teuthology.orchestra.run.smithi067.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 209 kB/s | 32 kB 00:00 2023-12-01T16:53:51.119 INFO:teuthology.orchestra.run.smithi067.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 539 kB/s | 17 kB 00:00 2023-12-01T16:53:51.161 INFO:teuthology.orchestra.run.smithi067.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 849 kB/s | 34 kB 00:00 2023-12-01T16:53:51.228 INFO:teuthology.orchestra.run.smithi067.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2023-12-01T16:53:51.387 INFO:teuthology.orchestra.run.smithi067.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 21 MB/s | 4.7 MB 00:00 2023-12-01T16:53:51.445 INFO:teuthology.orchestra.run.smithi067.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2023-12-01T16:53:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2023-12-01T16:53:51.621 INFO:teuthology.orchestra.run.smithi067.stdout:(8/12): libgfortran-11.4.1-2.3.el9.x86_64.rpm 5.3 MB/s | 803 kB 00:00 2023-12-01T16:53:51.805 INFO:teuthology.orchestra.run.smithi067.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2023-12-01T16:53:51.830 INFO:teuthology.orchestra.run.smithi067.stdout:(10/12): libquadmath-11.4.1-2.3.el9.x86_64.rpm 946 kB/s | 195 kB 00:00 2023-12-01T16:53:51.935 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-01T16:53:51.955 INFO:teuthology.orchestra.run.smithi067.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2023-12-01T16:53:52.038 INFO:teuthology.orchestra.run.smithi067.stdout:(12/12): ceph-mgr-diskprediction-local-18.0.0-7 6.7 MB/s | 7.4 MB 00:01 2023-12-01T16:53:52.041 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:52.042 INFO:teuthology.orchestra.run.smithi067.stdout:Total 32 MB/s | 37 MB 00:01 2023-12-01T16:53:52.055 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:53:52.081 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:53:52.082 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:53:52.496 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:53:52.497 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:53:52.945 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-01T16:53:52.967 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c 12/12 2023-12-01T16:53:53.221 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:53:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 1/12 2023-12-01T16:53:53.343 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 2/12 2023-12-01T16:53:53.440 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2023-12-01T16:53:53.480 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2023-12-01T16:53:53.666 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-01T16:53:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-01T16:53:53.773 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2023-12-01T16:53:54.451 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2023-12-01T16:53:54.560 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c 12/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c 1/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-01T16:53:55.531 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-01T16:53:55.532 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-01T16:53:55.532 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-01T16:53:55.532 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-01T16:53:55.532 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-01T16:53:55.848 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2023-12-01T16:53:55.918 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-01T16:53:55.918 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-01T16:53:55.919 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-01T16:53:55.920 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:55.920 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:53:56.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-rook 2023-12-01T16:53:56.881 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:15 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:53:56.931 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2023-12-01T16:53:56.960 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c 12/12 2023-12-01T16:53:57.029 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:53:57.030 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 50 k 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-01T16:53:57.031 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout:Install 13 Packages 2023-12-01T16:53:57.032 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:53:57.033 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 2.0 M 2023-12-01T16:53:57.033 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 26 M 2023-12-01T16:53:57.033 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:53:57.475 INFO:teuthology.orchestra.run.smithi092.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 188 kB/s | 18 kB 00:00 2023-12-01T16:53:57.525 INFO:teuthology.orchestra.run.smithi092.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 167 kB/s | 24 kB 00:00 2023-12-01T16:53:57.567 INFO:teuthology.orchestra.run.smithi092.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2023-12-01T16:53:57.592 INFO:teuthology.orchestra.run.smithi092.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-01T16:53:57.617 INFO:teuthology.orchestra.run.smithi092.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-01T16:53:57.643 INFO:teuthology.orchestra.run.smithi092.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 876 kB/s | 43 kB 00:00 2023-12-01T16:53:57.918 INFO:teuthology.orchestra.run.smithi092.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2023-12-01T16:53:57.951 INFO:teuthology.orchestra.run.smithi092.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 49 kB/s | 15 kB 00:00 2023-12-01T16:53:57.977 INFO:teuthology.orchestra.run.smithi092.stdout:(9/13): ceph-mgr-rook-18.0.0-7570.gd710c09d.el9 84 kB/s | 50 kB 00:00 2023-12-01T16:53:58.110 INFO:teuthology.orchestra.run.smithi092.stdout:(10/13): python3-google-auth-2.23.4-1.el9.noarc 1.0 MB/s | 197 kB 00:00 2023-12-01T16:53:58.194 INFO:teuthology.orchestra.run.smithi092.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2023-12-01T16:53:58.352 INFO:teuthology.orchestra.run.smithi092.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 157 kB/s | 59 kB 00:00 2023-12-01T16:53:58.511 INFO:teuthology.orchestra.run.smithi092.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2023-12-01T16:53:58.515 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:53:58.515 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-12-01T16:53:58.515 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:53:58.528 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:53:58.528 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:53:58.622 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:53:58.622 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:53:59.161 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:53:59.324 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-01T16:53:59.378 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-01T16:53:59.442 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-01T16:53:59.461 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c 12/12 2023-12-01T16:53:59.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c 1/12 2023-12-01T16:53:59.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2023-12-01T16:53:59.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2023-12-01T16:53:59.461 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2023-12-01T16:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2023-12-01T16:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2023-12-01T16:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2023-12-01T16:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2023-12-01T16:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2023-12-01T16:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 10/12 2023-12-01T16:53:59.462 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 11/12 2023-12-01T16:53:59.479 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-01T16:53:59.521 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-01T16:53:59.613 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-01T16:53:59.723 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-01T16:53:59.771 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-12-01T16:53:59.869 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-12-01T16:53:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-12-01T16:53:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-12-01T16:53:59.870 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:53:59.870 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:00.188 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-rook 2023-12-01T16:54:00.497 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-01T16:54:00.544 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-01T16:54:00.578 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-01T16:54:00.626 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-01T16:54:00.650 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 13/13 2023-12-01T16:54:00.795 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:15 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:00.950 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 50 k 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-12-01T16:54:00.951 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.23.4-1.el9 epel 197 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout:Install 13 Packages 2023-12-01T16:54:00.952 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:00.953 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 2.0 M 2023-12-01T16:54:00.953 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 26 M 2023-12-01T16:54:00.953 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:01.334 INFO:teuthology.orchestra.run.smithi067.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 177 kB/s | 24 kB 00:00 2023-12-01T16:54:01.359 INFO:teuthology.orchestra.run.smithi067.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2023-12-01T16:54:01.401 INFO:teuthology.orchestra.run.smithi067.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2023-12-01T16:54:01.427 INFO:teuthology.orchestra.run.smithi067.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2023-12-01T16:54:01.452 INFO:teuthology.orchestra.run.smithi067.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 13/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 1/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-01T16:54:01.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-01T16:54:01.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-01T16:54:01.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-01T16:54:01.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-01T16:54:01.477 INFO:teuthology.orchestra.run.smithi067.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 858 kB/s | 43 kB 00:00 2023-12-01T16:54:01.502 INFO:teuthology.orchestra.run.smithi067.stdout:(7/13): ceph-mgr-rook-18.0.0-7570.gd710c09d.el9 164 kB/s | 50 kB 00:00 2023-12-01T16:54:01.820 INFO:teuthology.orchestra.run.smithi067.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2023-12-01T16:54:01.856 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-01T16:54:01.856 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:01.856 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:01.856 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:01.857 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:01.945 INFO:teuthology.orchestra.run.smithi067.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2023-12-01T16:54:02.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-cephadm 2023-12-01T16:54:02.253 INFO:teuthology.orchestra.run.smithi067.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 192 kB/s | 59 kB 00:00 2023-12-01T16:54:02.270 INFO:teuthology.orchestra.run.smithi067.stdout:(11/13): python3-google-auth-2.23.4-1.el9.noarc 257 kB/s | 197 kB 00:00 2023-12-01T16:54:02.395 INFO:teuthology.orchestra.run.smithi067.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2023-12-01T16:54:02.429 INFO:teuthology.orchestra.run.smithi067.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 511 kB/s | 90 kB 00:00 2023-12-01T16:54:02.434 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:02.434 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-12-01T16:54:02.434 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:02.449 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:02.449 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:02.560 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:02.561 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:02.706 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:21 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:02.853 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 133 k 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-01T16:54:02.854 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-01T16:54:02.855 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-01T16:54:02.855 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:02.855 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:02.855 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:54:02.855 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2023-12-01T16:54:02.855 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:02.855 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 6.8 M 2023-12-01T16:54:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 33 M 2023-12-01T16:54:02.856 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:03.285 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2023-12-01T16:54:03.305 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.2 MB/s | 228 kB 00:00 2023-12-01T16:54:03.340 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2023-12-01T16:54:03.396 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2023-12-01T16:54:03.431 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2023-12-01T16:54:03.474 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2023-12-01T16:54:03.522 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 2.5 MB/s | 548 kB 00:00 2023-12-01T16:54:03.570 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2023-12-01T16:54:03.572 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2023-12-01T16:54:03.648 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 5.8 MB 00:00 2023-12-01T16:54:03.672 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 515 kB/s | 51 kB 00:00 2023-12-01T16:54:03.680 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.23.4-1.el9.noarch 7/13 2023-12-01T16:54:03.742 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2023-12-01T16:54:03.848 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): ceph-mgr-cephadm-18.0.0-7570.gd710c09d.e 183 kB/s | 133 kB 00:00 2023-12-01T16:54:03.851 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:03.852 INFO:teuthology.orchestra.run.smithi092.stdout:Total 6.8 MB/s | 6.8 MB 00:00 2023-12-01T16:54:03.852 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:03.861 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:03.861 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:03.948 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:03.949 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:04.252 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:04.412 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-01T16:54:04.466 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-01T16:54:04.510 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2023-12-01T16:54:04.570 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2023-12-01T16:54:04.605 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2023-12-01T16:54:04.653 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2023-12-01T16:54:04.682 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 13/13 2023-12-01T16:54:04.972 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-01T16:54:05.042 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-01T16:54:05.112 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-01T16:54:05.136 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 6/6 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 13/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 1/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.23.4-1.el9.noarch 10/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2023-12-01T16:54:05.483 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2023-12-01T16:54:05.845 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2023-12-01T16:54:05.845 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.23.4-1.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-12-01T16:54:05.846 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2023-12-01T16:54:05.847 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-12-01T16:54:05.847 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:05.847 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:05.988 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 6/6 2023-12-01T16:54:05.988 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/6 2023-12-01T16:54:05.988 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-01T16:54:05.988 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-01T16:54:05.988 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-01T16:54:05.988 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-01T16:54:06.090 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-cephadm 2023-12-01T16:54:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-01T16:54:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:06.304 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:06.304 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:54:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-01T16:54:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-01T16:54:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-01T16:54:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-01T16:54:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-01T16:54:06.305 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:06.305 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:06.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-fuse 2023-12-01T16:54:06.700 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:21 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:06.854 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 133 k 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:06.855 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:54:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:Install 6 Packages 2023-12-01T16:54:06.856 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 6.8 M 2023-12-01T16:54:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 33 M 2023-12-01T16:54:06.856 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:07.072 INFO:teuthology.orchestra.run.smithi067.stdout:(1/6): python3-jinja2-2.11.3-4.el9.noarch.rpm 1.4 MB/s | 228 kB 00:00 2023-12-01T16:54:07.129 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:26 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:07.138 INFO:teuthology.orchestra.run.smithi067.stdout:(2/6): ceph-mgr-cephadm-18.0.0-7570.gd710c09d.e 580 kB/s | 133 kB 00:00 2023-12-01T16:54:07.172 INFO:teuthology.orchestra.run.smithi067.stdout:(3/6): python3-asyncssh-2.13.2-3.el9.noarch.rpm 5.4 MB/s | 548 kB 00:00 2023-12-01T16:54:07.247 INFO:teuthology.orchestra.run.smithi067.stdout:(4/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2023-12-01T16:54:07.264 INFO:teuthology.orchestra.run.smithi067.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 469 kB/s | 58 kB 00:00 2023-12-01T16:54:07.275 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:07.276 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:07.276 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:54:07.276 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:07.276 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 885 k 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 963 k 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.8 M 2023-12-01T16:54:07.277 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:07.281 INFO:teuthology.orchestra.run.smithi067.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 472 kB/s | 51 kB 00:00 2023-12-01T16:54:07.282 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:07.283 INFO:teuthology.orchestra.run.smithi067.stdout:Total 16 MB/s | 6.8 MB 00:00 2023-12-01T16:54:07.283 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:07.292 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:07.292 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:07.388 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:07.389 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:07.407 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 604 kB/s | 79 kB 00:00 2023-12-01T16:54:07.607 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-fuse-18.0.0-7570.gd710c09d.el9.x86_ 2.6 MB/s | 885 kB 00:00 2023-12-01T16:54:07.608 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:07.609 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.8 MB/s | 963 kB 00:00 2023-12-01T16:54:07.609 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:07.629 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:07.630 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:07.688 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:07.708 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:07.709 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:07.853 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2023-12-01T16:54:07.900 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:07.917 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 2/6 2023-12-01T16:54:08.049 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-01T16:54:08.116 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 2/2 2023-12-01T16:54:08.475 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2023-12-01T16:54:08.563 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2023-12-01T16:54:08.631 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 5/6 2023-12-01T16:54:08.662 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 6/6 2023-12-01T16:54:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 2/2 2023-12-01T16:54:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/2 2023-12-01T16:54:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-01T16:54:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:09.148 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-01T16:54:09.148 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:09.149 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:09.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-volume 2023-12-01T16:54:09.438 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 6/6 2023-12-01T16:54:09.438 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 1/6 2023-12-01T16:54:09.438 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2023-12-01T16:54:09.439 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 3/6 2023-12-01T16:54:09.439 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 4/6 2023-12-01T16:54:09.439 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2023-12-01T16:54:09.749 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2023-12-01T16:54:09.749 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:09.749 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:09.749 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:54:09.749 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-12-01T16:54:09.749 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2023-12-01T16:54:09.749 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-12-01T16:54:09.750 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-12-01T16:54:09.750 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-12-01T16:54:09.750 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:09.750 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:09.975 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse 2023-12-01T16:54:09.993 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:28 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:10.144 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 262 k 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:10.145 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:54:10.146 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:10.146 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 262 k 2023-12-01T16:54:10.146 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.2 M 2023-12-01T16:54:10.146 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:10.528 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-volume-18.0.0-7570.gd710c09d.el9.noarch.rp 687 kB/s | 262 kB 00:00 2023-12-01T16:54:10.528 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:10.529 INFO:teuthology.orchestra.run.smithi092.stdout:Total 684 kB/s | 262 kB 00:00 2023-12-01T16:54:10.529 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:10.536 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:10.537 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:10.568 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:10.568 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:10.579 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:25 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:10.732 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 885 k 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:10.733 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2023-12-01T16:54:10.734 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:10.734 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 963 k 2023-12-01T16:54:10.734 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.8 M 2023-12-01T16:54:10.734 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:10.786 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:10.863 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:54:10.876 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 555 kB/s | 79 kB 00:00 2023-12-01T16:54:10.879 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:54:10.879 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:54:10.879 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-01T16:54:10.879 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:11.201 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-fuse-18.0.0-7570.gd710c09d.el9.x86_ 1.9 MB/s | 885 kB 00:00 2023-12-01T16:54:11.202 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:11.203 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.0 MB/s | 963 kB 00:00 2023-12-01T16:54:11.203 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:11.225 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:11.225 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:11.306 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:11.306 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:11.667 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2023-12-01T16:54:11.746 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 2/2 2023-12-01T16:54:11.866 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:54:11.866 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:11.866 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:11.867 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:54:11.867 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:11.867 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:12.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados-devel 2023-12-01T16:54:12.558 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 2/2 2023-12-01T16:54:12.559 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/2 2023-12-01T16:54:12.770 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:31 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:12.889 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2023-12-01T16:54:12.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:12.889 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:12.889 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2023-12-01T16:54:12.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:12.890 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:12.915 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:12.916 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:12.916 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:54:12.916 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:12.916 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:12.916 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 127 k 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 127 k 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 456 k 2023-12-01T16:54:12.917 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:13.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-volume 2023-12-01T16:54:13.232 INFO:teuthology.orchestra.run.smithi092.stdout:librados-devel-18.0.0-7570.gd710c09d.el9.x86_64 403 kB/s | 127 kB 00:00 2023-12-01T16:54:13.232 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:13.233 INFO:teuthology.orchestra.run.smithi092.stdout:Total 401 kB/s | 127 kB 00:00 2023-12-01T16:54:13.233 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:13.239 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:13.239 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:13.293 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:13.294 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:13.539 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:13.741 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:28 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:13.892 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:18.0.0-7570.gd710c09d.el9 ceph-noarch 262 k 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 262 k 2023-12-01T16:54:13.893 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.2 M 2023-12-01T16:54:13.894 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:13.976 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:14.139 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-volume-18.0.0-7570.gd710c09d.el9.noarch.rp 1.0 MB/s | 262 kB 00:00 2023-12-01T16:54:14.140 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:14.140 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.0 MB/s | 262 kB 00:00 2023-12-01T16:54:14.140 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:14.149 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:14.149 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:14.189 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:14.190 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:14.320 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:14.320 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:14.321 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:14.321 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:14.321 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:14.321 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:14.417 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:14.473 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:54:14.490 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:54:14.490 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:54:14.490 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-12-01T16:54:14.490 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:14.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs2 2023-12-01T16:54:15.151 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:34 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:15.251 INFO:teuthology.orchestra.run.smithi092.stdout:Package libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:15.301 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:15.302 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:54:15.302 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:15.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs-devel 2023-12-01T16:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 1/1 2023-12-01T16:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:18.0.0-7570.gd710c09d.el9.noarch 2023-12-01T16:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:15.515 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:15.818 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel 2023-12-01T16:54:15.950 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:34 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:16.097 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 31 k 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 31 k 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 139 k 2023-12-01T16:54:16.098 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:16.268 INFO:teuthology.orchestra.run.smithi092.stdout:libcephfs-devel-18.0.0-7570.gd710c09d.el9.x86_6 184 kB/s | 31 kB 00:00 2023-12-01T16:54:16.269 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:16.269 INFO:teuthology.orchestra.run.smithi092.stdout:Total 182 kB/s | 31 kB 00:00 2023-12-01T16:54:16.269 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:16.272 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:16.272 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:16.304 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:16.305 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:16.395 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:16.453 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:31 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:16.500 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:16.609 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:16.609 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:16.609 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 127 k 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 127 k 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 456 k 2023-12-01T16:54:16.610 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:16.895 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:16.959 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-18.0.0-7570.gd710c09d.el9.x86_64 364 kB/s | 127 kB 00:00 2023-12-01T16:54:16.960 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:16.960 INFO:teuthology.orchestra.run.smithi067.stdout:Total 362 kB/s | 127 kB 00:00 2023-12-01T16:54:16.960 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:16.966 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:16.966 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:17.025 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:17.025 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:17.160 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:17.160 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:17.161 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:17.161 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:17.182 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:17.304 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:17.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados2 2023-12-01T16:54:17.731 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:17.907 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:36 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:18.003 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:18.056 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:18.056 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:54:18.057 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:18.059 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:18.059 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:18.059 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:18.059 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:18.060 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:18.060 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:18.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librbd1 2023-12-01T16:54:18.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2 2023-12-01T16:54:18.693 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:37 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:18.789 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:18.840 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:18.841 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:54:18.841 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:18.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rados 2023-12-01T16:54:18.918 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:33 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:19.017 INFO:teuthology.orchestra.run.smithi067.stdout:Package libcephfs2-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:19.074 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:19.075 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:54:19.075 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:19.137 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs-devel 2023-12-01T16:54:19.486 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:38 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:19.581 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:19.631 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:19.632 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:54:19.632 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:19.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rgw 2023-12-01T16:54:19.748 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:34 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:19.900 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 31 k 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:19.901 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:54:19.902 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:19.902 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 31 k 2023-12-01T16:54:19.902 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 139 k 2023-12-01T16:54:19.902 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:20.137 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs-devel-18.0.0-7570.gd710c09d.el9.x86_6 132 kB/s | 31 kB 00:00 2023-12-01T16:54:20.138 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:20.138 INFO:teuthology.orchestra.run.smithi067.stdout:Total 132 kB/s | 31 kB 00:00 2023-12-01T16:54:20.138 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:20.141 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:20.141 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:20.176 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:20.177 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:20.260 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:20.267 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:39 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:20.362 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:20.371 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:20.412 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:20.413 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:54:20.413 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:20.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-cephfs 2023-12-01T16:54:20.753 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:21.024 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:21.057 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:40 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:21.153 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:21.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2 2023-12-01T16:54:21.204 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:21.205 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:54:21.205 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:21.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rbd 2023-12-01T16:54:21.787 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:36 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:21.845 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:40 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:21.885 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:21.939 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:21.940 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:21.940 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:54:21.940 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:21.989 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:21.990 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:54:21.990 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:22.002 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1 2023-12-01T16:54:22.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-fuse 2023-12-01T16:54:22.607 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:37 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:22.620 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:41 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:22.706 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:22.760 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:22.761 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:54:22.761 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:22.765 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:22.765 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:22.765 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 85 k 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:54:22.766 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:22.767 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 85 k 2023-12-01T16:54:22.767 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 231 k 2023-12-01T16:54:22.767 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:22.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados 2023-12-01T16:54:22.980 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-fuse-18.0.0-7570.gd710c09d.el9.x86_64.rpm 402 kB/s | 85 kB 00:00 2023-12-01T16:54:22.980 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:22.980 INFO:teuthology.orchestra.run.smithi092.stdout:Total 399 kB/s | 85 kB 00:00 2023-12-01T16:54:22.980 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:22.986 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:22.986 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:23.040 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:23.040 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:23.180 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:23.284 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:23.429 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:38 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:23.529 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rados-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:23.582 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:23.583 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:54:23.583 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:23.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw 2023-12-01T16:54:23.664 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:23.957 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:23.958 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:23.958 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:23.958 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:23.958 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:23.958 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:24.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-mirror 2023-12-01T16:54:24.259 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:39 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:24.358 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rgw-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:24.412 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:24.413 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:54:24.413 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:24.480 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs 2023-12-01T16:54:24.770 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:43 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:24.916 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:25.147 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 3.1 M 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:25.148 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 3.1 M 2023-12-01T16:54:25.149 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:40 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:25.149 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 13 M 2023-12-01T16:54:25.149 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:25.190 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-cephfs-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:25.244 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:25.245 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:54:25.245 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:25.310 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd 2023-12-01T16:54:25.315 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-mirror-18.0.0-7570.gd710c09d.el9.x86_64.rpm 7.8 MB/s | 3.1 MB 00:00 2023-12-01T16:54:25.316 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:25.316 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.7 MB/s | 3.1 MB 00:00 2023-12-01T16:54:25.316 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:25.325 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:25.325 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:25.397 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:25.397 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:25.819 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:25.883 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:25.907 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:25.907 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:54:25.907 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-01T16:54:25.907 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-01T16:54:25.907 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-01T16:54:25.908 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:25.915 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:40 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:26.013 INFO:teuthology.orchestra.run.smithi067.stdout:Package python3-rbd-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:54:26.066 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:26.067 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:54:26.068 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:26.127 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse 2023-12-01T16:54:26.731 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:41 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:26.884 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 85 k 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:54:26.885 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:26.886 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 85 k 2023-12-01T16:54:26.886 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 231 k 2023-12-01T16:54:26.886 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:27.072 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:27.072 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:27.072 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:27.072 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:27.072 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:27.072 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:27.206 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-18.0.0-7570.gd710c09d.el9.x86_64.rpm 266 kB/s | 85 kB 00:00 2023-12-01T16:54:27.207 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:27.207 INFO:teuthology.orchestra.run.smithi067.stdout:Total 265 kB/s | 85 kB 00:00 2023-12-01T16:54:27.207 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:27.215 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:27.215 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:27.272 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:27.273 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:27.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-nbd 2023-12-01T16:54:27.414 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:27.524 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:27.921 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:27.945 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:46 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 170 k 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:28.092 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:28.093 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:28.093 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:54:28.093 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:28.093 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 170 k 2023-12-01T16:54:28.093 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 490 k 2023-12-01T16:54:28.093 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:28.239 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:28.239 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:28.239 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:28.239 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:28.239 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:28.239 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:28.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror 2023-12-01T16:54:28.451 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-nbd-18.0.0-7570.gd710c09d.el9.x86_64.rpm 477 kB/s | 170 kB 00:00 2023-12-01T16:54:28.452 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:28.452 INFO:teuthology.orchestra.run.smithi092.stdout:Total 474 kB/s | 170 kB 00:00 2023-12-01T16:54:28.453 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:28.460 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:28.460 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:28.515 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:28.515 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:28.674 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:28.774 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:29.042 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:44 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:29.194 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 3.1 M 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:54:29.195 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:29.196 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 3.1 M 2023-12-01T16:54:29.196 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 13 M 2023-12-01T16:54:29.196 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:29.214 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:29.550 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:29.550 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:29.550 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:29.550 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:29.550 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:29.550 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:29.558 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-18.0.0-7570.gd710c09d.el9.x86_64.rpm 8.5 MB/s | 3.1 MB 00:00 2023-12-01T16:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout:Total 8.5 MB/s | 3.1 MB 00:00 2023-12-01T16:54:29.559 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:29.569 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:29.645 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:29.645 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:29.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base-debuginfo 2023-12-01T16:54:30.088 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:30.159 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:30.184 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:30.185 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-12-01T16:54:30.185 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-12-01T16:54:30.185 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-01T16:54:30.185 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-12-01T16:54:30.185 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:30.408 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:49 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:30.557 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:30.557 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 73 M 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 21 M 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2023-12-01T16:54:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:30.559 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 94 M 2023-12-01T16:54:30.559 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 513 M 2023-12-01T16:54:30.559 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:31.341 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:31.342 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:31.342 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:31.342 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:31.342 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:31.342 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:31.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd 2023-12-01T16:54:32.193 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:47 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:32.346 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 170 k 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:54:32.347 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:32.348 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 170 k 2023-12-01T16:54:32.348 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 490 k 2023-12-01T16:54:32.348 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:32.596 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): ceph-debuginfo-18.0.0-7570.gd710c09d.el9 10 MB/s | 21 MB 00:02 2023-12-01T16:54:32.627 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-18.0.0-7570.gd710c09d.el9.x86_64.rpm 611 kB/s | 170 kB 00:00 2023-12-01T16:54:32.628 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:32.628 INFO:teuthology.orchestra.run.smithi067.stdout:Total 607 kB/s | 170 kB 00:00 2023-12-01T16:54:32.628 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:32.637 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:32.637 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:32.695 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:32.695 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:32.861 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:32.953 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:33.370 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:33.661 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T16:54:33.662 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:33.662 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:33.662 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:33.662 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:33.662 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:33.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base-debuginfo 2023-12-01T16:54:34.488 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:49 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:34.646 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 73 M 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 21 M 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout:Install 2 Packages 2023-12-01T16:54:34.647 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 94 M 2023-12-01T16:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 513 M 2023-12-01T16:54:34.648 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:54:34.854 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-base-debuginfo-18.0.0-7570.gd710c09 17 MB/s | 73 MB 00:04 2023-12-01T16:54:34.856 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:34.856 INFO:teuthology.orchestra.run.smithi092.stdout:Total 22 MB/s | 94 MB 00:04 2023-12-01T16:54:34.857 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:54:34.861 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:54:34.861 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:54:35.472 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:54:35.472 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:54:36.713 INFO:teuthology.orchestra.run.smithi067.stdout:(1/2): ceph-debuginfo-18.0.0-7570.gd710c09d.el9 10 MB/s | 21 MB 00:02 2023-12-01T16:54:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:54:38.963 INFO:teuthology.orchestra.run.smithi067.stdout:(2/2): ceph-base-debuginfo-18.0.0-7570.gd710c09 17 MB/s | 73 MB 00:04 2023-12-01T16:54:38.965 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:54:38.965 INFO:teuthology.orchestra.run.smithi067.stdout:Total 22 MB/s | 94 MB 00:04 2023-12-01T16:54:38.966 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:54:38.971 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:54:38.971 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:54:39.610 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:54:39.610 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:54:42.784 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:54:46.059 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/2 2023-12-01T16:54:46.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 2/2 2023-12-01T16:54:49.558 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 2/2 2023-12-01T16:54:49.559 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/2 2023-12-01T16:54:49.835 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2/2 2023-12-01T16:54:49.835 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:49.835 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:54:49.835 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:49.835 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:49.835 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:49.835 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:54:50.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-common-debuginfo 2023-12-01T16:54:50.644 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:02:09 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:54:50.789 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 404 M 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:54:50.795 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:54:50.796 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:54:50.796 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 404 M 2023-12-01T16:54:50.796 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.6 G 2023-12-01T16:54:50.796 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:54:50.802 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/2 2023-12-01T16:54:50.875 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 2/2 2023-12-01T16:54:54.355 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 2/2 2023-12-01T16:54:54.355 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/2 2023-12-01T16:54:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2/2 2023-12-01T16:54:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:54.623 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:54:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:54:54.623 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:54.623 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:54:54.808 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-common-debuginfo 2023-12-01T16:54:55.422 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:02:10 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:54:55.587 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:54:55.588 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:55.634 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:54:55.634 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:55.634 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:54:55.634 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 404 M 2023-12-01T16:54:55.634 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:55.635 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:54:55.635 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:54:55.635 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:54:55.635 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:54:55.635 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 404 M 2023-12-01T16:54:55.635 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.6 G 2023-12-01T16:54:55.635 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:55:15.434 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-common-debuginfo-18.0.0-7570.gd710c09d.el9 20 MB/s | 404 MB 00:19 2023-12-01T16:55:15.434 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:55:15.435 INFO:teuthology.orchestra.run.smithi067.stdout:Total 20 MB/s | 404 MB 00:19 2023-12-01T16:55:15.435 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:55:15.439 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:55:15.439 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:55:17.960 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:55:17.960 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:55:28.701 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-common-debuginfo-18.0.0-7570.gd710c09d.el9 11 MB/s | 404 MB 00:37 2023-12-01T16:55:28.701 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:55:28.702 INFO:teuthology.orchestra.run.smithi092.stdout:Total 11 MB/s | 404 MB 00:37 2023-12-01T16:55:28.702 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:55:28.706 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:55:28.706 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:55:31.100 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:55:31.101 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:56:01.174 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:56:01.374 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T16:56:09.026 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T16:56:09.295 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T16:56:09.295 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:09.295 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:56:09.295 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:56:09.295 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:09.295 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:56:09.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-01T16:56:10.198 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:25 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:56:10.350 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.4 M 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================== 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 1.4 M 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 6.4 M 2023-12-01T16:56:10.351 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:56:10.917 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-75 2.5 MB/s | 1.4 MB 00:00 2023-12-01T16:56:10.917 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:56:10.918 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.5 MB/s | 1.4 MB 00:00 2023-12-01T16:56:10.918 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:56:10.922 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:56:10.922 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:56:10.967 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:56:10.967 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:56:11.236 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:56:11.335 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7570. 1/1 2023-12-01T16:56:11.760 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7570. 1/1 2023-12-01T16:56:11.927 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:56:12.034 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7570. 1/1 2023-12-01T16:56:12.034 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:12.034 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:56:12.034 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:56:12.034 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:12.034 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:56:12.067 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T16:56:12.193 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-01T16:56:12.796 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:03:27 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:56:12.949 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 173 M 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 173 M 2023-12-01T16:56:12.950 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 731 M 2023-12-01T16:56:12.951 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:56:19.777 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T16:56:20.058 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T16:56:20.058 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:20.058 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:56:20.058 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:56:20.059 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:20.059 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:56:20.300 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-12-01T16:56:20.893 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:03:39 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:56:21.048 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.4 M 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 1.4 M 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 6.4 M 2023-12-01T16:56:21.049 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:56:21.336 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-radosgw-debuginfo-18.0.0-7570.gd710c09d.el 21 MB/s | 173 MB 00:08 2023-12-01T16:56:21.337 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:56:21.337 INFO:teuthology.orchestra.run.smithi067.stdout:Total 21 MB/s | 173 MB 00:08 2023-12-01T16:56:21.338 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:56:21.342 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:56:21.342 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:56:21.679 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-immutable-object-cache-debuginfo-18.0.0-75 2.3 MB/s | 1.4 MB 00:00 2023-12-01T16:56:21.679 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:56:21.679 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.3 MB/s | 1.4 MB 00:00 2023-12-01T16:56:21.680 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:56:21.683 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:56:21.683 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:56:21.726 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:56:21.726 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:56:21.978 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:56:22.094 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.0.0-7570. 1/1 2023-12-01T16:56:22.438 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:56:22.438 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:56:22.511 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.0.0-7570. 1/1 2023-12-01T16:56:22.787 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.0.0-7570. 1/1 2023-12-01T16:56:22.787 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:22.787 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:56:22.787 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:56:22.788 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:22.788 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:56:22.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw-debuginfo 2023-12-01T16:56:23.527 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:03:42 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:56:23.671 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 173 M 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:56:23.672 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:23.673 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 173 M 2023-12-01T16:56:23.673 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 731 M 2023-12-01T16:56:23.673 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:56:32.014 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-radosgw-debuginfo-18.0.0-7570.gd710c09d.el 21 MB/s | 173 MB 00:08 2023-12-01T16:56:32.015 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:56:32.015 INFO:teuthology.orchestra.run.smithi092.stdout:Total 21 MB/s | 173 MB 00:08 2023-12-01T16:56:32.015 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:56:32.019 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:56:32.019 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:56:33.067 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:56:33.067 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:56:41.224 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:56:41.326 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x 1/1 2023-12-01T16:56:45.757 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x 1/1 2023-12-01T16:56:46.047 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x 1/1 2023-12-01T16:56:46.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:46.047 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:56:46.047 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:56:46.047 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:46.047 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:56:46.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-test-debuginfo 2023-12-01T16:56:46.811 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:04:01 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:56:46.961 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 676 M 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:56:46.962 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 676 M 2023-12-01T16:56:46.963 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 3.1 G 2023-12-01T16:56:46.963 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:56:50.821 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:56:50.908 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x 1/1 2023-12-01T16:56:55.393 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x 1/1 2023-12-01T16:56:55.666 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x 1/1 2023-12-01T16:56:55.666 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:55.666 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:56:55.667 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:56:55.667 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:55.667 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:56:55.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-test-debuginfo 2023-12-01T16:56:56.436 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:04:15 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:56:56.579 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:56:56.580 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:56:56.580 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:56:56.580 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:56:56.580 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:56:56.580 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 676 M 2023-12-01T16:56:56.580 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:56.580 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:56:56.581 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:56:56.581 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:56:56.581 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:56:56.581 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 676 M 2023-12-01T16:56:56.581 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 3.1 G 2023-12-01T16:56:56.581 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:57:35.264 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-test-debuginfo-18.0.0-7570.gd710c09d.el9.x 17 MB/s | 676 MB 00:38 2023-12-01T16:57:35.265 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:57:35.265 INFO:teuthology.orchestra.run.smithi092.stdout:Total 17 MB/s | 676 MB 00:38 2023-12-01T16:57:35.265 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:57:35.269 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:57:35.269 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:57:39.266 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:57:39.266 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:57:56.489 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-test-debuginfo-18.0.0-7570.gd710c09d.el9.x 9.7 MB/s | 676 MB 01:09 2023-12-01T16:57:56.490 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:57:56.491 INFO:teuthology.orchestra.run.smithi067.stdout:Total 9.7 MB/s | 676 MB 01:09 2023-12-01T16:57:56.491 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:57:56.496 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:57:56.497 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:58:00.684 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:58:00.684 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:58:50.375 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:58:50.474 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T16:58:59.675 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T16:58:59.946 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T16:58:59.946 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:58:59.946 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:58:59.946 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:58:59.946 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:58:59.946 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:59:00.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base-debuginfo 2023-12-01T16:59:00.808 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:19 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:59:00.903 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:59:00.951 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:59:00.953 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-12-01T16:59:00.953 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:59:01.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-debuginfo 2023-12-01T16:59:01.590 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:20 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:59:01.733 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 19 M 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 19 M 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 101 M 2023-12-01T16:59:01.734 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:59:03.070 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-mgr-debuginfo-18.0.0-7570.gd710c09d.el9.x8 14 MB/s | 19 MB 00:01 2023-12-01T16:59:03.070 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:59:03.070 INFO:teuthology.orchestra.run.smithi092.stdout:Total 14 MB/s | 19 MB 00:01 2023-12-01T16:59:03.070 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:59:03.073 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:59:03.073 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:59:03.218 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:59:03.218 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:59:05.402 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:59:05.516 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:06.480 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:06.778 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:06.778 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:06.778 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:59:06.778 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:59:06.778 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:06.778 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:59:06.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mds-debuginfo 2023-12-01T16:59:07.522 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:26 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:59:07.665 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 46 M 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:59:07.666 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:07.667 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 46 M 2023-12-01T16:59:07.667 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 168 M 2023-12-01T16:59:07.667 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:59:09.887 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-mds-debuginfo-18.0.0-7570.gd710c09d.el9.x8 21 MB/s | 46 MB 00:02 2023-12-01T16:59:09.887 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:59:09.888 INFO:teuthology.orchestra.run.smithi092.stdout:Total 21 MB/s | 46 MB 00:02 2023-12-01T16:59:09.888 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:59:09.895 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:59:09.895 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:59:10.197 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:59:10.198 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:59:14.796 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:59:14.884 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:16.149 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:59:16.251 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T16:59:16.253 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:16.526 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:59:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:59:16.526 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:16.527 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:59:16.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mon-debuginfo 2023-12-01T16:59:17.271 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:36 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:59:17.413 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:59:17.416 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:17.416 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 111 M 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 111 M 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 506 M 2023-12-01T16:59:17.417 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:59:22.488 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-mon-debuginfo-18.0.0-7570.gd710c09d.el9.x8 22 MB/s | 111 MB 00:05 2023-12-01T16:59:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:59:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:Total 22 MB/s | 111 MB 00:05 2023-12-01T16:59:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:59:22.493 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:59:22.493 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:59:23.177 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:59:23.179 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T16:59:23.678 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T16:59:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T16:59:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:23.943 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:59:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:59:23.943 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:23.943 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:59:24.196 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-base-debuginfo 2023-12-01T16:59:24.799 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:06:39 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:59:24.898 INFO:teuthology.orchestra.run.smithi067.stdout:Package ceph-base-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 is already installed. 2023-12-01T16:59:24.950 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:59:24.951 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-12-01T16:59:24.951 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:59:25.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mgr-debuginfo 2023-12-01T16:59:25.607 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:06:40 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:59:25.758 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 19 M 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:59:25.759 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:25.760 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 19 M 2023-12-01T16:59:25.760 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 101 M 2023-12-01T16:59:25.760 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:59:26.844 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-mgr-debuginfo-18.0.0-7570.gd710c09d.el9.x8 18 MB/s | 19 MB 00:01 2023-12-01T16:59:26.845 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:59:26.845 INFO:teuthology.orchestra.run.smithi067.stdout:Total 18 MB/s | 19 MB 00:01 2023-12-01T16:59:26.846 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:59:26.850 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:59:26.850 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:59:27.003 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:59:27.004 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:59:29.305 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:59:29.394 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:30.413 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:30.680 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:59:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:59:30.680 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:30.681 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:59:30.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mds-debuginfo 2023-12-01T16:59:31.438 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:06:46 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:59:31.588 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:59:31.589 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:31.589 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:59:31.589 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:31.589 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:59:31.589 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 46 M 2023-12-01T16:59:31.589 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:31.590 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:59:31.590 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:31.590 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:59:31.590 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:31.590 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 46 M 2023-12-01T16:59:31.590 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 168 M 2023-12-01T16:59:31.590 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:59:33.831 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-mds-debuginfo-18.0.0-7570.gd710c09d.el9.x8 21 MB/s | 46 MB 00:02 2023-12-01T16:59:33.831 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:59:33.832 INFO:teuthology.orchestra.run.smithi067.stdout:Total 21 MB/s | 46 MB 00:02 2023-12-01T16:59:33.832 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:59:33.836 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:59:33.837 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:59:34.156 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:59:34.156 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:59:34.752 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T16:59:34.841 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:38.014 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:38.287 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T16:59:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:59:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:38.287 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T16:59:38.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-osd-debuginfo 2023-12-01T16:59:39.017 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T16:59:39.063 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:06:58 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T16:59:39.115 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:39.207 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T16:59:39.207 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 311 M 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 311 M 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.3 G 2023-12-01T16:59:39.208 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T16:59:40.435 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T16:59:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:40.716 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T16:59:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T16:59:40.716 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:40.716 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T16:59:40.864 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-mon-debuginfo 2023-12-01T16:59:41.463 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:06:56 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T16:59:41.613 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T16:59:41.614 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:41.614 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 111 M 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 111 M 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 506 M 2023-12-01T16:59:41.615 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T16:59:55.015 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-mon-debuginfo-18.0.0-7570.gd710c09d.el9.x8 8.3 MB/s | 111 MB 00:13 2023-12-01T16:59:55.016 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:59:55.016 INFO:teuthology.orchestra.run.smithi067.stdout:Total 8.3 MB/s | 111 MB 00:13 2023-12-01T16:59:55.017 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T16:59:55.021 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T16:59:55.021 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T16:59:55.741 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T16:59:55.741 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T16:59:56.726 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-osd-debuginfo-18.0.0-7570.gd710c09d.el9.x8 18 MB/s | 311 MB 00:17 2023-12-01T16:59:56.727 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T16:59:56.727 INFO:teuthology.orchestra.run.smithi092.stdout:Total 18 MB/s | 311 MB 00:17 2023-12-01T16:59:56.728 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T16:59:56.731 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T16:59:56.732 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T16:59:58.577 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T16:59:58.577 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:00:08.070 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:00:08.174 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:00:11.382 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:00:11.673 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:00:11.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:00:11.673 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:00:11.673 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:00:11.673 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:00:11.673 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:00:11.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-osd-debuginfo 2023-12-01T17:00:12.454 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:07:27 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:00:12.605 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 311 M 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 311 M 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.3 G 2023-12-01T17:00:12.606 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:00:26.141 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-osd-debuginfo-18.0.0-7570.gd710c09d.el9.x8 23 MB/s | 311 MB 00:13 2023-12-01T17:00:26.142 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:00:26.142 INFO:teuthology.orchestra.run.smithi067.stdout:Total 23 MB/s | 311 MB 00:13 2023-12-01T17:00:26.146 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:00:26.147 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:00:26.147 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:00:28.092 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:00:28.092 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:00:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:00:30.668 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:00:38.889 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:00:39.160 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:00:39.160 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:39.160 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:00:39.160 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:00:39.160 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:39.160 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:00:39.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-fuse-debuginfo 2023-12-01T17:00:39.957 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:07:58 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:00:40.102 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:00:40.103 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:00:40.103 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:40.103 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:00:40.103 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 5.9 M 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 5.9 M 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 28 M 2023-12-01T17:00:40.104 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:00:40.772 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-fuse-debuginfo-18.0.0-7570.gd710c09d.el9.x 8.8 MB/s | 5.9 MB 00:00 2023-12-01T17:00:40.773 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:00:40.773 INFO:teuthology.orchestra.run.smithi092.stdout:Total 8.8 MB/s | 5.9 MB 00:00 2023-12-01T17:00:40.773 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:00:40.777 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:00:40.777 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:00:40.843 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:00:40.844 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:00:41.574 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:00:41.687 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:00:42.226 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:00:42.507 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:00:42.507 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:42.507 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:00:42.508 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:00:42.508 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:42.508 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:00:42.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados-devel-debuginfo 2023-12-01T17:00:43.271 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:02 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:00:43.417 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:00:43.417 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:43.417 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 108 k 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:00:43.418 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:43.419 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 108 k 2023-12-01T17:00:43.419 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 440 k 2023-12-01T17:00:43.419 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:00:43.641 INFO:teuthology.orchestra.run.smithi092.stdout:librados-devel-debuginfo-18.0.0-7570.gd710c09d. 487 kB/s | 108 kB 00:00 2023-12-01T17:00:43.642 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:00:43.642 INFO:teuthology.orchestra.run.smithi092.stdout:Total 484 kB/s | 108 kB 00:00 2023-12-01T17:00:43.642 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:00:43.645 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:00:43.646 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:00:43.679 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:00:43.679 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:00:43.791 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:00:43.881 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:00:44.269 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:00:44.547 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:00:44.547 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:44.547 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:00:44.547 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:00:44.547 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:44.547 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:00:44.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs2-debuginfo 2023-12-01T17:00:45.298 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:04 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:00:45.442 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 5.3 M 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 5.3 M 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 24 M 2023-12-01T17:00:45.443 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:00:46.119 INFO:teuthology.orchestra.run.smithi092.stdout:libcephfs2-debuginfo-18.0.0-7570.gd710c09d.el9. 7.8 MB/s | 5.3 MB 00:00 2023-12-01T17:00:46.120 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:00:46.120 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.8 MB/s | 5.3 MB 00:00 2023-12-01T17:00:46.121 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:00:46.124 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:00:46.124 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:00:46.189 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:00:46.189 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:00:46.850 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:00:46.960 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:00:47.470 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:00:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:00:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:47.754 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:00:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:00:47.754 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:47.755 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:00:47.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados2-debuginfo 2023-12-01T17:00:48.524 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:07 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:00:48.667 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:00:48.668 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:48.668 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:00:48.668 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:48.668 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:00:48.668 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 44 M 2023-12-01T17:00:48.668 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:48.669 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:00:48.669 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:48.669 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:00:48.669 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:48.669 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 44 M 2023-12-01T17:00:48.669 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 250 M 2023-12-01T17:00:48.669 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:00:50.805 INFO:teuthology.orchestra.run.smithi092.stdout:librados2-debuginfo-18.0.0-7570.gd710c09d.el9.x 20 MB/s | 44 MB 00:02 2023-12-01T17:00:50.806 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:00:50.806 INFO:teuthology.orchestra.run.smithi092.stdout:Total 20 MB/s | 44 MB 00:02 2023-12-01T17:00:50.806 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:00:50.810 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:00:50.810 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:00:51.106 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:00:51.106 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:00:56.070 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:00:56.168 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:00:58.017 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:00:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:00:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:58.279 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:00:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:00:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:58.279 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:00:58.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librbd1-debuginfo 2023-12-01T17:00:59.078 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:18 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:00:59.237 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:00:59.237 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 51 M 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 51 M 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 305 M 2023-12-01T17:00:59.238 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:01.857 INFO:teuthology.orchestra.run.smithi092.stdout:librbd1-debuginfo-18.0.0-7570.gd710c09d.el9.x86 20 MB/s | 51 MB 00:02 2023-12-01T17:01:01.858 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:01.858 INFO:teuthology.orchestra.run.smithi092.stdout:Total 19 MB/s | 51 MB 00:02 2023-12-01T17:01:01.858 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:01.862 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:01.862 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:01.917 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:02.023 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:02.210 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:02.211 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:08.140 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:08.246 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:10.166 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:10.417 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:10.459 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:10.459 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:10.460 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:10.460 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:10.655 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-fuse-debuginfo 2023-12-01T17:01:10.684 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:10.684 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:10.684 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:10.684 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:10.684 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:10.684 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:10.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-cephfs-debuginfo 2023-12-01T17:01:11.259 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:08:26 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:11.409 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:11.410 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:11.410 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:11.410 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:11.410 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:11.410 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 5.9 M 2023-12-01T17:01:11.410 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:11.411 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:11.411 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:11.411 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:11.411 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:11.411 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 5.9 M 2023-12-01T17:01:11.411 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 28 M 2023-12-01T17:01:11.411 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:11.483 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:30 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 615 k 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:11.629 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:01:11.630 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:11.630 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:01:11.630 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:11.630 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 615 k 2023-12-01T17:01:11.631 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.1 M 2023-12-01T17:01:11.631 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:11.925 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-fuse-debuginfo-18.0.0-7570.gd710c09d.el9.x 11 MB/s | 5.9 MB 00:00 2023-12-01T17:01:11.925 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:11.926 INFO:teuthology.orchestra.run.smithi067.stdout:Total 11 MB/s | 5.9 MB 00:00 2023-12-01T17:01:11.926 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:11.929 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:11.930 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:11.984 INFO:teuthology.orchestra.run.smithi092.stdout:python3-cephfs-debuginfo-18.0.0-7570.gd710c09d. 1.7 MB/s | 615 kB 00:00 2023-12-01T17:01:11.984 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:11.985 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.7 MB/s | 615 kB 00:00 2023-12-01T17:01:11.985 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:11.988 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:11.988 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:12.001 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:12.001 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:12.025 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:12.025 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:12.196 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:12.299 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:12.746 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:12.774 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:12.909 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:01:13.034 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:13.034 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:13.035 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:13.035 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:13.035 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:13.035 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:13.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rados-debuginfo 2023-12-01T17:01:13.486 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:01:13.755 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:01:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:13.756 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:13.756 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:13.756 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:13.852 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:32 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:01:13.918 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados-devel-debuginfo 2023-12-01T17:01:13.997 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:01:13.999 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:13.999 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:01:13.999 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:13.999 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:01:13.999 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 769 k 2023-12-01T17:01:13.999 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:13.999 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:01:14.000 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:14.000 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:01:14.000 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:14.000 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 769 k 2023-12-01T17:01:14.000 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.7 M 2023-12-01T17:01:14.000 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:14.298 INFO:teuthology.orchestra.run.smithi092.stdout:python3-rados-debuginfo-18.0.0-7570.gd710c09d.e 2.5 MB/s | 769 kB 00:00 2023-12-01T17:01:14.298 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:14.298 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.5 MB/s | 769 kB 00:00 2023-12-01T17:01:14.298 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:14.299 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:14.299 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:14.337 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:14.337 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:14.518 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:14.522 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:08:29 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:14.619 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9. 1/1 2023-12-01T17:01:14.675 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:14.676 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:14.676 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:14.676 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:14.676 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:14.676 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 108 k 2023-12-01T17:01:14.676 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:14.676 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:14.677 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:14.677 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:14.677 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:14.677 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 108 k 2023-12-01T17:01:14.677 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 440 k 2023-12-01T17:01:14.677 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:15.002 INFO:teuthology.orchestra.run.smithi067.stdout:librados-devel-debuginfo-18.0.0-7570.gd710c09d. 333 kB/s | 108 kB 00:00 2023-12-01T17:01:15.003 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:15.003 INFO:teuthology.orchestra.run.smithi067.stdout:Total 332 kB/s | 108 kB 00:00 2023-12-01T17:01:15.003 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:15.007 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:15.007 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:15.012 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9. 1/1 2023-12-01T17:01:15.043 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:15.043 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:15.161 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:15.261 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:15.300 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9. 1/1 2023-12-01T17:01:15.300 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:15.301 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:15.301 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:15.301 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:15.301 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:15.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rbd-debuginfo 2023-12-01T17:01:15.645 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:15.927 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:15.927 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:15.927 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:15.927 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:15.927 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:15.928 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:16.106 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:35 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:01:16.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install libcephfs2-debuginfo 2023-12-01T17:01:16.249 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:01:16.250 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:16.250 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:01:16.250 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:16.250 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.1 M 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 1.1 M 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 4.3 M 2023-12-01T17:01:16.251 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:16.711 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:08:31 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:16.748 INFO:teuthology.orchestra.run.smithi092.stdout:python3-rbd-debuginfo-18.0.0-7570.gd710c09d.el9 2.3 MB/s | 1.1 MB 00:00 2023-12-01T17:01:16.748 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:16.748 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.3 MB/s | 1.1 MB 00:00 2023-12-01T17:01:16.748 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:16.752 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:16.752 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:16.791 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:16.791 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:16.862 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:16.862 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 5.3 M 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 5.3 M 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 24 M 2023-12-01T17:01:16.863 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:17.007 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:17.105 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:17.504 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:17.547 INFO:teuthology.orchestra.run.smithi067.stdout:libcephfs2-debuginfo-18.0.0-7570.gd710c09d.el9. 7.8 MB/s | 5.3 MB 00:00 2023-12-01T17:01:17.548 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:17.548 INFO:teuthology.orchestra.run.smithi067.stdout:Total 7.7 MB/s | 5.3 MB 00:00 2023-12-01T17:01:17.549 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:17.552 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:17.552 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:17.620 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:17.620 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:17.782 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:17.782 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:17.782 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:17.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rgw-debuginfo 2023-12-01T17:01:18.317 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:18.415 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:01:18.558 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:37 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:01:18.701 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 337 k 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 337 k 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.0 M 2023-12-01T17:01:18.702 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:19.090 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:01:19.126 INFO:teuthology.orchestra.run.smithi092.stdout:python3-rgw-debuginfo-18.0.0-7570.gd710c09d.el9 798 kB/s | 337 kB 00:00 2023-12-01T17:01:19.126 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:19.126 INFO:teuthology.orchestra.run.smithi092.stdout:Total 795 kB/s | 337 kB 00:00 2023-12-01T17:01:19.126 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:19.130 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:19.130 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:19.164 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:19.165 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:19.299 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:19.369 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:01:19.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:19.370 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:19.370 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:19.370 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:19.407 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:19.661 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librados2-debuginfo 2023-12-01T17:01:19.780 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:20.055 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:20.055 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:20.055 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:20.055 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:20.055 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:20.055 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:20.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-fuse-debuginfo 2023-12-01T17:01:20.267 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:08:35 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:20.420 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:20.420 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 44 M 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 44 M 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 250 M 2023-12-01T17:01:20.421 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:20.827 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:39 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:01:20.970 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:01:20.971 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:20.971 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:01:20.971 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 508 k 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 508 k 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.5 M 2023-12-01T17:01:20.972 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:21.520 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-fuse-debuginfo-18.0.0-7570.gd710c09d.el9.x8 928 kB/s | 508 kB 00:00 2023-12-01T17:01:21.520 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:21.521 INFO:teuthology.orchestra.run.smithi092.stdout:Total 926 kB/s | 508 kB 00:00 2023-12-01T17:01:21.521 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:21.524 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:21.524 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:21.562 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:21.562 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:21.721 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:21.817 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:22.170 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:22.440 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:22.440 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:22.440 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:22.440 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:22.440 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:22.440 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:22.642 INFO:teuthology.orchestra.run.smithi067.stdout:librados2-debuginfo-18.0.0-7570.gd710c09d.el9.x 20 MB/s | 44 MB 00:02 2023-12-01T17:01:22.642 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:Total 20 MB/s | 44 MB 00:02 2023-12-01T17:01:22.643 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:22.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-mirror-debuginfo 2023-12-01T17:01:22.648 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:22.648 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:22.962 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:22.962 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:23.217 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:42 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:01:23.361 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 51 M 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 51 M 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 296 M 2023-12-01T17:01:23.362 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:25.789 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-mirror-debuginfo-18.0.0-7570.gd710c09d.el9. 21 MB/s | 51 MB 00:02 2023-12-01T17:01:25.790 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:25.790 INFO:teuthology.orchestra.run.smithi092.stdout:Total 21 MB/s | 51 MB 00:02 2023-12-01T17:01:25.791 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:25.794 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:25.795 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:26.126 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:26.126 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:28.235 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:28.326 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:01:30.180 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:01:30.452 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_ 1/1 2023-12-01T17:01:30.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:30.452 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:30.452 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:30.452 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:30.452 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:30.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install librbd1-debuginfo 2023-12-01T17:01:31.262 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:08:46 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:31.413 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:31.413 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:31.413 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 51 M 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:31.414 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:31.415 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 51 M 2023-12-01T17:01:31.415 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 305 M 2023-12-01T17:01:31.415 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:31.982 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:32.086 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:01:34.036 INFO:teuthology.orchestra.run.smithi067.stdout:librbd1-debuginfo-18.0.0-7570.gd710c09d.el9.x86 19 MB/s | 51 MB 00:02 2023-12-01T17:01:34.037 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:34.037 INFO:teuthology.orchestra.run.smithi067.stdout:Total 19 MB/s | 51 MB 00:02 2023-12-01T17:01:34.038 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:34.042 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:34.042 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:34.097 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:01:34.387 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:01:34.387 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:34.387 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:34.387 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:34.387 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:34.387 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:34.402 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:34.402 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:34.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-nbd-debuginfo 2023-12-01T17:01:35.156 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:08:54 ago on Fri 01 Dec 2023 04:52:41 PM UTC. 2023-12-01T17:01:35.301 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-12-01T17:01:35.301 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:35.301 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2023-12-01T17:01:35.301 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.2 M 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:35.302 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 1.2 M 2023-12-01T17:01:35.303 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 5.4 M 2023-12-01T17:01:35.303 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-12-01T17:01:35.790 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-nbd-debuginfo-18.0.0-7570.gd710c09d.el9.x86 2.5 MB/s | 1.2 MB 00:00 2023-12-01T17:01:35.790 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:35.791 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.5 MB/s | 1.2 MB 00:00 2023-12-01T17:01:35.791 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-12-01T17:01:35.794 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-12-01T17:01:35.795 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-12-01T17:01:35.835 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-12-01T17:01:35.835 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-12-01T17:01:36.067 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-12-01T17:01:36.180 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:36.585 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:36.853 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:36.853 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:36.853 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-12-01T17:01:36.854 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:36.854 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:01:36.854 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-12-01T17:01:37.067 DEBUG:teuthology.parallel:result is None 2023-12-01T17:01:40.677 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:40.778 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:42.981 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:43.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:01:43.265 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:43.265 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:43.265 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:43.265 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:43.265 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:43.454 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-cephfs-debuginfo 2023-12-01T17:01:44.060 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:08:59 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 615 k 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:44.213 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:44.214 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:44.214 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:44.214 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 615 k 2023-12-01T17:01:44.214 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.1 M 2023-12-01T17:01:44.214 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:44.480 INFO:teuthology.orchestra.run.smithi067.stdout:python3-cephfs-debuginfo-18.0.0-7570.gd710c09d. 2.3 MB/s | 615 kB 00:00 2023-12-01T17:01:44.481 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:44.481 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.2 MB/s | 615 kB 00:00 2023-12-01T17:01:44.481 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:44.484 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:44.485 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:44.524 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:44.524 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:44.694 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:44.806 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:45.176 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:45.484 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9 1/1 2023-12-01T17:01:45.484 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:45.484 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:45.484 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:45.485 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:45.485 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:45.670 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rados-debuginfo 2023-12-01T17:01:46.274 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:09:01 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:46.427 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:46.428 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:46.428 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:46.428 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 769 k 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 769 k 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.7 M 2023-12-01T17:01:46.429 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:46.725 INFO:teuthology.orchestra.run.smithi067.stdout:python3-rados-debuginfo-18.0.0-7570.gd710c09d.e 2.5 MB/s | 769 kB 00:00 2023-12-01T17:01:46.726 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:46.726 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.5 MB/s | 769 kB 00:00 2023-12-01T17:01:46.726 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:46.730 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:46.731 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:46.771 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:46.771 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:46.961 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:47.076 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9. 1/1 2023-12-01T17:01:47.469 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9. 1/1 2023-12-01T17:01:47.747 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9. 1/1 2023-12-01T17:01:47.747 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:47.747 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:47.747 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:47.747 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:47.747 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:47.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rbd-debuginfo 2023-12-01T17:01:48.525 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:09:03 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:48.676 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:48.677 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:48.677 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:48.677 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:48.677 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:48.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.1 M 2023-12-01T17:01:48.677 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:48.677 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:48.678 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:48.678 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:48.678 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:48.678 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 1.1 M 2023-12-01T17:01:48.678 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 4.3 M 2023-12-01T17:01:48.678 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:48.978 INFO:teuthology.orchestra.run.smithi067.stdout:python3-rbd-debuginfo-18.0.0-7570.gd710c09d.el9 3.8 MB/s | 1.1 MB 00:00 2023-12-01T17:01:48.978 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:48.979 INFO:teuthology.orchestra.run.smithi067.stdout:Total 3.7 MB/s | 1.1 MB 00:00 2023-12-01T17:01:48.979 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:48.982 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:48.982 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:49.024 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:49.025 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:49.251 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:49.354 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:49.738 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:50.021 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:50.021 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:50.021 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:50.021 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:50.021 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:50.022 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:50.210 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install python3-rgw-debuginfo 2023-12-01T17:01:50.810 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:09:05 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:50.960 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 337 k 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:50.961 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:50.962 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 337 k 2023-12-01T17:01:50.962 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 1.0 M 2023-12-01T17:01:50.962 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:51.206 INFO:teuthology.orchestra.run.smithi067.stdout:python3-rgw-debuginfo-18.0.0-7570.gd710c09d.el9 1.4 MB/s | 337 kB 00:00 2023-12-01T17:01:51.207 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:51.207 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.3 MB/s | 337 kB 00:00 2023-12-01T17:01:51.207 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:51.211 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:51.211 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:51.249 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:51.249 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:51.391 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:51.507 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:51.915 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:52.189 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x8 1/1 2023-12-01T17:01:52.189 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:52.189 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:52.189 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:52.189 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:52.189 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:52.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-fuse-debuginfo 2023-12-01T17:01:52.947 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:09:07 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:53.097 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:53.098 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:53.098 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:53.098 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 508 k 2023-12-01T17:01:53.098 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:53.098 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:53.099 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:53.099 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:53.099 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:53.099 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 508 k 2023-12-01T17:01:53.099 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 2.5 M 2023-12-01T17:01:53.099 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:53.376 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-fuse-debuginfo-18.0.0-7570.gd710c09d.el9.x8 1.8 MB/s | 508 kB 00:00 2023-12-01T17:01:53.377 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:53.377 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.8 MB/s | 508 kB 00:00 2023-12-01T17:01:53.377 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:53.380 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:53.381 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:53.420 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:53.420 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:01:53.589 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:01:53.695 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:54.104 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:54.397 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_6 1/1 2023-12-01T17:01:54.397 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:54.397 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:01:54.398 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:01:54.398 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:54.398 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:01:54.567 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-mirror-debuginfo 2023-12-01T17:01:55.167 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:09:10 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:01:55.319 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:01:55.320 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:55.320 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:01:55.320 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:55.320 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:01:55.320 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 51 M 2023-12-01T17:01:55.321 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:55.321 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:01:55.321 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:01:55.321 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:01:55.321 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:01:55.321 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 51 M 2023-12-01T17:01:55.322 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 296 M 2023-12-01T17:01:55.322 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:01:57.746 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-mirror-debuginfo-18.0.0-7570.gd710c09d.el9. 21 MB/s | 51 MB 00:02 2023-12-01T17:01:57.747 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:01:57.747 INFO:teuthology.orchestra.run.smithi067.stdout:Total 21 MB/s | 51 MB 00:02 2023-12-01T17:01:57.747 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:01:57.751 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:01:57.752 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:01:58.101 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:01:58.102 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:02:04.295 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:02:04.397 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:02:06.427 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:02:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86 1/1 2023-12-01T17:02:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:06.713 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:02:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:02:06.713 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:06.713 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:02:06.879 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install rbd-nbd-debuginfo 2023-12-01T17:02:07.492 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:09:22 ago on Fri 01 Dec 2023 04:52:45 PM UTC. 2023-12-01T17:02:07.644 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repo Size 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-debuginfo x86_64 2:18.0.0-7570.gd710c09d.el9 ceph 1.2 M 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout:Install 1 Package 2023-12-01T17:02:07.645 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:07.646 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 1.2 M 2023-12-01T17:02:07.646 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 5.4 M 2023-12-01T17:02:07.646 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-12-01T17:02:08.231 INFO:teuthology.orchestra.run.smithi067.stdout:rbd-nbd-debuginfo-18.0.0-7570.gd710c09d.el9.x86 2.1 MB/s | 1.2 MB 00:00 2023-12-01T17:02:08.232 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-12-01T17:02:08.232 INFO:teuthology.orchestra.run.smithi067.stdout:Total 2.1 MB/s | 1.2 MB 00:00 2023-12-01T17:02:08.232 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-12-01T17:02:08.235 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-12-01T17:02:08.235 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-12-01T17:02:08.279 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-12-01T17:02:08.279 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-12-01T17:02:08.534 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-12-01T17:02:08.632 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:02:09.033 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:02:09.313 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 1/1 2023-12-01T17:02:09.313 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:09.313 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-12-01T17:02:09.313 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-debuginfo-2:18.0.0-7570.gd710c09d.el9.x86_64 2023-12-01T17:02:09.313 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:09.313 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-12-01T17:02:09.470 DEBUG:teuthology.parallel:result is None 2023-12-01T17:02:09.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d710c09d2ffe0f0858468330fd3d69f1f4506222 2023-12-01T17:02:09.761 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-01T17:02:09.782 INFO:teuthology.orchestra.run.smithi067.stdout:18.0.0-7570.gd710c09d.el9 2023-12-01T17:02:09.783 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7570.gd710c09d.el9 2023-12-01T17:02:09.783 INFO:teuthology.task.install:The correct ceph version 18.0.0-7570.gd710c09d is installed. 2023-12-01T17:02:09.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d710c09d2ffe0f0858468330fd3d69f1f4506222 2023-12-01T17:02:10.053 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-01T17:02:10.077 INFO:teuthology.orchestra.run.smithi092.stdout:18.0.0-7570.gd710c09d.el9 2023-12-01T17:02:10.078 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7570.gd710c09d.el9 2023-12-01T17:02:10.078 INFO:teuthology.task.install:The correct ceph version 18.0.0-7570.gd710c09d is installed. 2023-12-01T17:02:10.078 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-01T17:02:10.078 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:10.078 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-01T17:02:10.110 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:10.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-01T17:02:10.149 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-01T17:02:10.149 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:10.149 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2023-12-01T17:02:10.182 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-01T17:02:10.253 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:10.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2023-12-01T17:02:10.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-01T17:02:10.354 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-01T17:02:10.355 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:10.355 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-01T17:02:10.386 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-01T17:02:10.457 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:10.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-01T17:02:10.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-01T17:02:10.555 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-01T17:02:10.556 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:10.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2023-12-01T17:02:10.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-01T17:02:10.658 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:10.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2023-12-01T17:02:10.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-01T17:02:10.757 INFO:teuthology.run_tasks:Running task ceph... 2023-12-01T17:02:10.831 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-01T17:02:10.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2023-12-01T17:02:10.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2023-12-01T17:02:10.863 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-01T17:02:10.863 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-01T17:02:10.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-01T17:02:10.934 INFO:tasks.ceph:Creating extra log directories... 2023-12-01T17:02:10.934 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-01T17:02:10.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-01T17:02:11.017 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-12-01T17:02:11.017 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-01T17:02:11.017 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838', 'branch': 'wip-neorados-learning-from-experience', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7473838', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-neorados-learning-from-experience'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-neorados-learning-from-experience', 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 5183, 'sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-neorados-learning-from-experience', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'd710c09d2ffe0f0858468330fd3d69f1f4506222', 'targets': {'smithi067.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBQIyZoIHRylwtXjzYm+hE6V8vXmABu3fn7ivKqnzNMIizqP9FIVhl3+6s6v1db7RCg8qcA69TIPmuMquZ12nzg=', 'smithi092.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsY9SWMXphNX4his3hpk3d0Ii08nhsOTfGtAq7TpsCy1q9isqLbgjSEt+no/DLci5rxXXU1Cwoatw1jkwPA7Qw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'f2de108beb0a6dde7f3e6c96039071828a1f6269', 'timestamp': '2023-12-01_15:09:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-01T17:02:11.018 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-01T17:02:11.050 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-12-01T17:02:11.065 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2023-12-01T17:02:11.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2023-12-01T17:02:11.134 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:11.134 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2023-12-01T17:02:11.190 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-01T17:02:11.190 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2023-12-01T17:02:11.245 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 662 Links: 1 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-12-01 16:54:30.457415856 +0000 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-12-01 16:51:51.543905818 +0000 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-12-01 16:51:51.543905818 +0000 2023-12-01T17:02:11.246 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2023-12-01 16:51:51.543905818 +0000 2023-12-01T17:02:11.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-01T17:02:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-12-01T17:02:11.315 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-12-01T17:02:11.316 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000202867 s, 2.5 MB/s 2023-12-01T17:02:11.316 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-01T17:02:11.373 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 684 Links: 1 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-12-01 16:54:30.457415856 +0000 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-12-01 16:51:52.046894703 +0000 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-12-01 16:51:52.046894703 +0000 2023-12-01T17:02:11.428 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2023-12-01 16:51:52.046894703 +0000 2023-12-01T17:02:11.429 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-01T17:02:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-12-01T17:02:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-12-01T17:02:11.497 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00017265 s, 3.0 MB/s 2023-12-01T17:02:11.499 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-01T17:02:11.556 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-12-01 16:54:30.457415856 +0000 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-12-01 16:51:52.555883455 +0000 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-12-01 16:51:52.555883455 +0000 2023-12-01T17:02:11.611 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2023-12-01 16:51:52.555883455 +0000 2023-12-01T17:02:11.612 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-01T17:02:11.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-12-01T17:02:11.681 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-12-01T17:02:11.681 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.00022756 s, 2.2 MB/s 2023-12-01T17:02:11.682 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-01T17:02:11.739 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2023-12-01T17:02:11.795 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-01T17:02:11.795 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:11.796 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 722 Links: 1 2023-12-01T17:02:11.796 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:11.796 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:11.796 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-12-01 16:54:30.458415834 +0000 2023-12-01T17:02:11.796 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-12-01 16:51:53.053872451 +0000 2023-12-01T17:02:11.796 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-12-01 16:51:53.053872451 +0000 2023-12-01T17:02:11.796 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2023-12-01 16:51:53.053872451 +0000 2023-12-01T17:02:11.796 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-01T17:02:11.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-12-01T17:02:11.865 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-12-01T17:02:11.865 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000185507 s, 2.8 MB/s 2023-12-01T17:02:11.866 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-01T17:02:11.923 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:11.923 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:11.923 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2023-12-01T17:02:11.941 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-01T17:02:11.941 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2023-12-01T17:02:11.998 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 664 Links: 1 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-12-01 16:54:26.156352014 +0000 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-12-01 16:51:43.361777602 +0000 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-12-01 16:51:43.361777602 +0000 2023-12-01T17:02:11.999 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-12-01 16:51:43.361777602 +0000 2023-12-01T17:02:11.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-01T17:02:12.067 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-12-01T17:02:12.067 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-12-01T17:02:12.067 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000219825 s, 2.3 MB/s 2023-12-01T17:02:12.068 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-01T17:02:12.126 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2023-12-01T17:02:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-01T17:02:12.182 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:12.183 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 682 Links: 1 2023-12-01T17:02:12.183 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:12.183 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:12.183 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-12-01 16:54:26.156352014 +0000 2023-12-01T17:02:12.183 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-12-01 16:51:43.860763737 +0000 2023-12-01T17:02:12.183 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-12-01 16:51:43.860763737 +0000 2023-12-01T17:02:12.183 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-12-01 16:51:43.860763737 +0000 2023-12-01T17:02:12.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-01T17:02:12.249 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-12-01T17:02:12.249 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-12-01T17:02:12.249 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000200998 s, 2.5 MB/s 2023-12-01T17:02:12.251 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-01T17:02:12.307 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2023-12-01T17:02:12.363 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-01T17:02:12.363 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:12.363 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 701 Links: 1 2023-12-01T17:02:12.363 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:12.363 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:12.363 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-12-01 16:54:26.156352014 +0000 2023-12-01T17:02:12.363 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-12-01 16:51:44.359749873 +0000 2023-12-01T17:02:12.364 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-12-01 16:51:44.359749873 +0000 2023-12-01T17:02:12.364 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-12-01 16:51:44.359749873 +0000 2023-12-01T17:02:12.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-01T17:02:12.430 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-12-01T17:02:12.430 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-12-01T17:02:12.431 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000162271 s, 3.2 MB/s 2023-12-01T17:02:12.432 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-01T17:02:12.488 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 719 Links: 1 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-12-01 16:54:26.157351987 +0000 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-12-01 16:51:44.861735926 +0000 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-12-01 16:51:44.861735926 +0000 2023-12-01T17:02:12.543 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2023-12-01 16:51:44.861735926 +0000 2023-12-01T17:02:12.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-01T17:02:12.608 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-12-01T17:02:12.609 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-12-01T17:02:12.609 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000168345 s, 3.0 MB/s 2023-12-01T17:02:12.610 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-01T17:02:12.666 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:12.666 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-01T17:02:12.666 INFO:tasks.ceph:Generating config... 2023-12-01T17:02:12.667 INFO:tasks.ceph:[client] debug monc = 20 2023-12-01T17:02:12.667 INFO:tasks.ceph:[client] debug ms = 1 2023-12-01T17:02:12.667 INFO:tasks.ceph:[client] debug objecter = 20 2023-12-01T17:02:12.667 INFO:tasks.ceph:[client] debug rados = 20 2023-12-01T17:02:12.667 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-01T17:02:12.667 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-12-01T17:02:12.667 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-01T17:02:12.668 INFO:tasks.ceph:[global] ms type = async 2023-12-01T17:02:12.668 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-12-01T17:02:12.668 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-12-01T17:02:12.669 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] debug monc = 20 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-01T17:02:12.669 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd class default list = * 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd class load list = * 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-12-01T17:02:12.670 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-12-01T17:02:12.671 INFO:tasks.ceph:Setting up mon.a... 2023-12-01T17:02:12.671 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-12-01T17:02:12.732 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2023-12-01T17:02:12.734 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-12-01T17:02:12.785 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-01T17:02:12.816 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.c', '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]'), ('mon.b', '172.21.15.92')] 2023-12-01T17:02:12.817 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.92', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-12-01T17:02:12.818 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:12.818 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-01T17:02:12.874 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --addv c '[v2:172.21.15.67:3301,v1:172.21.15.67:6790]' --add b 172.21.15.92 --print /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:12.960 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:12.960 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:02:12.960 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = pacific 2023-12-01T17:02:12.960 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2023-12-01T17:02:12.960 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 16 (pacific) 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:02:12.961 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-12-01T17:02:12.962 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-01T17:02:12.977 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2c621237-d4b8-4504-931d-6a9429822e96... 2023-12-01T17:02:12.978 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-01T17:02:13.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2023-12-01T17:02:13.073 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2023-12-01T17:02:13.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon_allow_pool_size_one = true 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.075 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.92" 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon client directed command retry = 5 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon election default strategy = 1 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: ms inject socket failures = 5000 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: ms type = async 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: osd heartbeat grace = 80 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.076 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock profile = high_recovery_ops 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug trim objects = true 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: bdev async discard = True 2023-12-01T17:02:13.078 INFO:teuthology.orchestra.run.smithi092.stdout: bdev enable discard = True 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore allocator = bitmap 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore block size = 96636764160 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore fsck on mount = True 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore zero block detection = True 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluefs = 1 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluestore = 1 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: debug monc = 20 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: debug rocksdb = 10 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd backfillfull_ratio = 0.85 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd full ratio = 0.9 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd nearfull ratio = 0.8 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: osd class default list = * 2023-12-01T17:02:13.079 INFO:teuthology.orchestra.run.smithi092.stdout: osd class load list = * 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd client watch timeout = 120 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify cached snaps = True 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify missing on start = True 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd failsafe full ratio = 0.95 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd fast shutdown = False 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock override recovery settings = True 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = bluestore 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd op queue = debug_random 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd op queue cut off = debug_random 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd shutdown pgref assert = True 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2023-12-01T17:02:13.080 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2023-12-01T17:02:13.081 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: # 1m isn't quite enough 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: mon_down_mkfs_grace = 2m 2023-12-01T17:02:13.082 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_filestore_osds = false 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon min osdmap epochs = 50 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd crush smoke test = False 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune interval = 2 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune min = 15 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon osdmap full prune txsize = 2 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon scrub interval = 300 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on pool no app = False 2023-12-01T17:02:13.083 INFO:teuthology.orchestra.run.smithi092.stdout: paxos service trim min = 10 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: debug monc = 20 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: debug objecter = 20 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout: debug rados = 20 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2023-12-01T17:02:13.084 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2023-12-01T17:02:13.085 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2023-12-01T17:02:13.086 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2023-12-01T17:02:13.087 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,[v2:172.21.15.67:3301,v1:172.21.15.67:6790],172.21.15.92" 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: mon client directed command retry = 5 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: mon election default strategy = 1 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: ms inject socket failures = 5000 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: ms type = async 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: osd heartbeat grace = 80 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2023-12-01T17:02:13.088 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug trim objects = true 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.089 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2023-12-01T17:02:13.090 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore zero block detection = True 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 10 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2023-12-01T17:02:13.091 INFO:teuthology.orchestra.run.smithi067.stdout: osd class default list = * 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd class load list = * 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd client watch timeout = 120 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify cached snaps = True 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify missing on start = True 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 0.95 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd fast shutdown = False 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock override recovery settings = True 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = bluestore 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue = debug_random 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd op queue cut off = debug_random 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: osd shutdown pgref assert = True 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2023-12-01T17:02:13.092 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2023-12-01T17:02:13.093 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2023-12-01T17:02:13.094 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon min osdmap epochs = 50 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd crush smoke test = False 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune interval = 2 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune min = 15 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon osdmap full prune txsize = 2 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon scrub interval = 300 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on pool no app = False 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: paxos service trim min = 10 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-01T17:02:13.095 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-01T17:02:13.096 INFO:teuthology.orchestra.run.smithi067.stdout: debug monc = 20 2023-12-01T17:02:13.096 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-12-01T17:02:13.096 INFO:teuthology.orchestra.run.smithi067.stdout: debug objecter = 20 2023-12-01T17:02:13.096 INFO:teuthology.orchestra.run.smithi067.stdout: debug rados = 20 2023-12-01T17:02:13.096 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2023-12-01T17:02:13.096 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2023-12-01T17:02:13.096 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2023-12-01T17:02:13.097 INFO:tasks.ceph:Creating admin key on mon.a... 2023-12-01T17:02:13.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-12-01T17:02:13.198 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-01T17:02:13.198 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:13.198 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-12-01T17:02:13.215 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:13.215 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-12-01T17:02:13.270 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:02:13.270 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:13.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-01T17:02:13.270 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-01T17:02:13.358 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:13.358 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:13.413 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:02:13.413 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:13.413 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2023-12-01T17:02:13.413 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-12-01T17:02:13.457 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:13.457 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:13.512 INFO:tasks.ceph:Setting up mon nodes... 2023-12-01T17:02:13.512 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-01T17:02:13.513 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-12-01T17:02:13.576 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-12-01T17:02:13.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-12-01T17:02:13.646 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-12-01T17:02:13.648 INFO:tasks.ceph:Setting up mds nodes... 2023-12-01T17:02:13.648 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-01T17:02:13.648 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-12-01T17:02:13.696 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-12-01T17:02:13.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-12-01T17:02:13.757 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-12-01T17:02:13.772 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-01T17:02:13.772 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-12-01T17:02:13.773 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-12-01T17:02:13.803 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:13.804 INFO:tasks.ceph:role: osd.0 2023-12-01T17:02:13.804 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:02:13.804 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-01T17:02:13.883 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:13.883 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:13.883 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:13.884 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:13.884 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:13.884 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:13.884 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:14.402 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-12-01T17:02:14.404 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-12-01T17:02:14.405 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-12-01T17:02:14.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-12-01T17:02:14.594 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-12-01T17:02:14.666 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:14.666 INFO:tasks.ceph:role: osd.1 2023-12-01T17:02:14.666 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:02:14.666 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:14.744 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:14.745 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:14.745 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:15.190 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-12-01T17:02:15.192 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-12-01T17:02:15.192 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-12-01T17:02:15.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-12-01T17:02:15.328 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-12-01T17:02:15.400 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:15.400 INFO:tasks.ceph:role: osd.2 2023-12-01T17:02:15.400 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:02:15.401 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-01T17:02:15.478 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:15.479 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:15.967 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-12-01T17:02:15.969 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-12-01T17:02:15.969 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-12-01T17:02:16.134 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-12-01T17:02:16.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-12-01T17:02:16.242 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:16.243 INFO:tasks.ceph:role: osd.3 2023-12-01T17:02:16.243 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:02:16.243 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:16.322 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:16.788 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-12-01T17:02:16.790 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-12-01T17:02:16.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-12-01T17:02:16.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-12-01T17:02:16.976 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:17.084+0000 7fb0dc748500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-12-01T17:02:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:17.084+0000 7fb0dc748500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-12-01T17:02:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:17.084+0000 7fb0dc748500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-01T17:02:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:17.084+0000 7fb0dc748500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-01T17:02:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:17.084+0000 7fb0dc748500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-12-01T17:02:17.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:17.085+0000 7fb0dc748500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-12-01T17:02:19.927 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-12-01T17:02:19.964 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:20.060+0000 7f0160a16500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-12-01T17:02:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:20.060+0000 7f0160a16500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-12-01T17:02:20.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:20.060+0000 7f0160a16500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-01T17:02:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:20.060+0000 7f0160a16500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-01T17:02:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:20.060+0000 7f0160a16500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-12-01T17:02:20.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:20.061+0000 7f0160a16500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-12-01T17:02:22.923 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-12-01T17:02:22.959 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:23.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:23.054+0000 7f87ede0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-12-01T17:02:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:23.054+0000 7f87ede0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-12-01T17:02:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:23.054+0000 7f87ede0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-01T17:02:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:23.054+0000 7f87ede0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-01T17:02:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:23.054+0000 7f87ede0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-12-01T17:02:23.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:23.054+0000 7f87ede0e500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-12-01T17:02:25.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-12-01T17:02:25.957 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:26.053+0000 7f721201d500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-12-01T17:02:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:26.053+0000 7f721201d500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-12-01T17:02:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:26.053+0000 7f721201d500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-01T17:02:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:26.053+0000 7f721201d500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-01T17:02:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:26.053+0000 7f721201d500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-12-01T17:02:26.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:26.054+0000 7f721201d500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-12-01T17:02:28.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-12-01T17:02:28.928 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-12-01T17:02:28.958 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:28.959 INFO:tasks.ceph:role: osd.4 2023-12-01T17:02:28.959 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:02:28.959 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:29.035 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:29.548 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-12-01T17:02:29.550 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-12-01T17:02:29.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-12-01T17:02:29.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-12-01T17:02:29.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-12-01T17:02:29.801 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:29.801 INFO:tasks.ceph:role: osd.5 2023-12-01T17:02:29.801 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:02:29.801 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:29.877 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:29.878 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:30.329 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-12-01T17:02:30.331 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-12-01T17:02:30.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-12-01T17:02:30.424 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-12-01T17:02:30.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-12-01T17:02:30.532 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:30.532 INFO:tasks.ceph:role: osd.6 2023-12-01T17:02:30.532 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:02:30.533 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:30.612 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:31.173 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-12-01T17:02:31.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-12-01T17:02:31.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-12-01T17:02:31.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-12-01T17:02:31.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-12-01T17:02:31.432 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-12-01T17:02:31.432 INFO:tasks.ceph:role: osd.7 2023-12-01T17:02:31.432 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:02:31.432 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-12-01T17:02:31.508 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-12-01T17:02:31.508 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-01T17:02:31.508 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-01T17:02:31.508 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-12-01T17:02:31.508 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-01T17:02:31.508 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-12-01T17:02:31.508 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-01T17:02:31.509 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-12-01T17:02:31.509 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-01T17:02:31.509 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-01T17:02:32.020 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-12-01T17:02:32.022 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-12-01T17:02:32.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-12-01T17:02:32.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-12-01T17:02:32.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:32.262 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:32.262+0000 7fb8ce669500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-12-01T17:02:32.263 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:32.263+0000 7fb8ce669500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-12-01T17:02:32.263 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:32.263+0000 7fb8ce669500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-01T17:02:32.263 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:32.263+0000 7fb8ce669500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-01T17:02:32.263 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:32.263+0000 7fb8ce669500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-12-01T17:02:32.263 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:32.263+0000 7fb8ce669500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-12-01T17:02:35.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-12-01T17:02:35.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:35.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:35.227+0000 7ff444614500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-12-01T17:02:35.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:35.227+0000 7ff444614500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-12-01T17:02:35.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:35.227+0000 7ff444614500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-01T17:02:35.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:35.227+0000 7ff444614500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-01T17:02:35.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:35.227+0000 7ff444614500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-12-01T17:02:35.228 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:35.228+0000 7ff444614500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-12-01T17:02:38.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-12-01T17:02:38.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:38.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:38.208+0000 7f8e1bebe500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-12-01T17:02:38.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:38.208+0000 7f8e1bebe500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-12-01T17:02:38.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:38.208+0000 7f8e1bebe500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-01T17:02:38.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:38.208+0000 7f8e1bebe500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-01T17:02:38.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:38.208+0000 7f8e1bebe500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-12-01T17:02:38.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:38.209+0000 7f8e1bebe500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-12-01T17:02:41.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-12-01T17:02:41.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:41.165 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:41.164+0000 7fb42e218500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-12-01T17:02:41.165 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:41.164+0000 7fb42e218500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-12-01T17:02:41.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:41.164+0000 7fb42e218500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-01T17:02:41.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:41.164+0000 7fb42e218500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-01T17:02:41.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:41.164+0000 7fb42e218500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-12-01T17:02:41.166 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:02:41.165+0000 7fb42e218500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-12-01T17:02:44.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-12-01T17:02:44.049 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-01T17:02:44.049 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:44.049 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-12-01T17:02:44.086 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:44.086 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-12-01T17:02:44.161 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:44.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-12-01T17:02:44.234 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:44.234 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-12-01T17:02:44.306 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:44.306 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-12-01T17:02:44.379 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:44.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-12-01T17:02:44.410 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:44.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-12-01T17:02:44.478 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:44.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-12-01T17:02:44.547 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:44.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-12-01T17:02:44.614 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:44.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-12-01T17:02:44.682 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:02:44.683 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-12-01T17:02:44.700 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:02:44.700 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-12-01T17:02:44.738 INFO:tasks.ceph:Adding keys to all mons... 2023-12-01T17:02:44.738 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-01T17:02:44.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2023-12-01T17:02:44.781 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCVEWplvBRoIhAAN9PmF7t+wYftMGEWQjbmEA== 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCZEWplBkcWBRAAOoRwiz8eVAMGKwH9gPx4Qw== 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCcEWplaAymAxAASPuIxOZV3+uqQdpZ8xGntg== 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCfEWplrtlEAxAAFg19D2AhaIoyHSyqegHrQA== 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCiEWplzOs5AxAAUKN44glBTP3HUtbCmOMllQ== 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCVEWpl99mKJhAAG5yRvkcKPofV/cJt9iu/8g== 2023-12-01T17:02:44.782 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCoEWplojmvDxAAXqzoMEttcvWBmbfybV5EcQ== 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCrEWplFOmPDRAANLUV2WRNE+F2tgwR1VLcEw== 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCuEWplgcRxDBAARKYA7Tvv4+dUoTA3z+39UQ== 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCxEWplvGPjCRAAK3Nczx1ua4f105eEh1zMzw== 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCVEWpljxKDKRAAoZvUNiEykwZ9xZWEbyNONw== 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2023-12-01T17:02:44.783 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQCVEWplU6QwLRAATRLtniO9LOTHRRBy7MhEwg== 2023-12-01T17:02:44.804 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2023-12-01T17:02:44.804 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCVEWplvBRoIhAAN9PmF7t+wYftMGEWQjbmEA== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCZEWplBkcWBRAAOoRwiz8eVAMGKwH9gPx4Qw== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCcEWplaAymAxAASPuIxOZV3+uqQdpZ8xGntg== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCfEWplrtlEAxAAFg19D2AhaIoyHSyqegHrQA== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCiEWplzOs5AxAAUKN44glBTP3HUtbCmOMllQ== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCVEWpl99mKJhAAG5yRvkcKPofV/cJt9iu/8g== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCoEWplojmvDxAAXqzoMEttcvWBmbfybV5EcQ== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCrEWplFOmPDRAANLUV2WRNE+F2tgwR1VLcEw== 2023-12-01T17:02:44.805 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2023-12-01T17:02:44.806 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCuEWplgcRxDBAARKYA7Tvv4+dUoTA3z+39UQ== 2023-12-01T17:02:44.806 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2023-12-01T17:02:44.806 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCxEWplvGPjCRAAK3Nczx1ua4f105eEh1zMzw== 2023-12-01T17:02:44.806 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2023-12-01T17:02:44.806 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCVEWpljxKDKRAAoZvUNiEykwZ9xZWEbyNONw== 2023-12-01T17:02:44.806 INFO:teuthology.orchestra.run.smithi092.stdout:[client.1] 2023-12-01T17:02:44.806 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQCVEWplU6QwLRAATRLtniO9LOTHRRBy7MhEwg== 2023-12-01T17:02:44.807 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-01T17:02:44.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-01T17:02:44.900 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:44.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:44.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:44.960 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.014 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.017 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.070 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.125 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-01T17:02:45.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-01T17:02:45.180 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.236 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-01T17:02:45.402 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-01T17:02:45.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-01T17:02:45.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-01T17:02:45.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-01T17:02:45.520 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-01T17:02:45.521 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-12-01T17:02:45.551 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-01T17:02:45.652 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-12-01T17:02:45.683 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-12-01T17:02:45.753 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-01T17:02:45.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-12-01T17:02:45.884 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-12-01T17:02:45.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-12-01T17:02:46.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-12-01T17:02:46.035 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:46.038 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-12-01T17:02:46.090 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-12-01T17:02:46.090 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-01T17:02:46.091 INFO:tasks.ceph.mon.a:Restarting daemon 2023-12-01T17:02:46.091 DEBUG:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-12-01T17:02:46.094 INFO:tasks.ceph.mon.a:Started 2023-12-01T17:02:46.094 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-01T17:02:46.094 INFO:tasks.ceph.mon.c:Restarting daemon 2023-12-01T17:02:46.094 DEBUG:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-12-01T17:02:46.096 INFO:tasks.ceph.mon.c:Started 2023-12-01T17:02:46.096 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-12-01T17:02:46.096 INFO:tasks.ceph.mon.b:Restarting daemon 2023-12-01T17:02:46.096 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-12-01T17:02:46.133 INFO:tasks.ceph.mon.b:Started 2023-12-01T17:02:46.133 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-12-01T17:02:46.133 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-12-01T17:02:46.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-12-01T17:02:46.135 INFO:tasks.ceph.mgr.y:Started 2023-12-01T17:02:46.135 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-12-01T17:02:46.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-12-01T17:02:46.137 INFO:tasks.ceph.mgr.x:Started 2023-12-01T17:02:46.137 INFO:tasks.ceph:Setting crush tunables to default 2023-12-01T17:02:46.138 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.246+0000 7f16fdb9e640 10 monclient: get_monmap_and_config 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.246+0000 7f16fdb9e640 10 monclient: build_initial_monmap 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.246+0000 7f16fdb9e640 10 monclient: monmap: 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:46.247257+0000 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:46.247257+0000 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:02:46.250 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:02:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:02:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:02:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:02:46.251 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:02:46.251 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:02:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.247+0000 7f16fdb9e640 1 Processor -- start 2023-12-01T17:02:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.247+0000 7f16fdb9e640 1 -- start start 2023-12-01T17:02:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.247+0000 7f16fdb9e640 10 monclient: init 2023-12-01T17:02:46.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.247+0000 7f16fdb9e640 10 monclient: _reopen_session rank -1 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.247+0000 7f16fdb9e640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.247+0000 7f16fdb9e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 0x7f16f80a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 10 monclient(hunting): picked mon.noname-a con 0x7f16f80a8990 addr v2:172.21.15.67:3300/0 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16f80b2210 addr v1:172.21.15.67:6789/0 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 0x7f16f813f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 10 monclient(hunting): picked mon.noname-c con 0x7f16f813f1b0 addr v2:172.21.15.92:3300/0 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80aa910 con 0x7f16f80b2210 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f8145640 con 0x7f16f80a8990 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:02:46.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f81461e0 con 0x7f16f813f1b0 2023-12-01T17:02:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 10 monclient(hunting): _renew_subs 2023-12-01T17:02:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16fdb9e640 10 monclient(hunting): authenticate will time out at 1297.485501s 2023-12-01T17:02:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16f77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 msgr2=0x7f16f80a8d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-12-01T17:02:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16f6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16f80b2210 legacy=0x7f16f8145ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-12-01T17:02:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 0x7f16f80a8d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:02:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 msgr2=0x7f16f813f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:46.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.248+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 0x7f16f813f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:02:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.449+0000 7f16f77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 msgr2=0x7f16f80a8d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-12-01T17:02:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.449+0000 7f16f6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16f80b2210 legacy=0x7f16f8145ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-12-01T17:02:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.449+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 0x7f16f80a8d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:02:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.449+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 msgr2=0x7f16f813f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:46.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.449+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 0x7f16f813f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:02:46.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.850+0000 7f16f6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16f80b2210 legacy=0x7f16f8145ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-12-01T17:02:46.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.850+0000 7f16f77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 msgr2=0x7f16f80a8d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-12-01T17:02:46.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.850+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 0x7f16f80a8d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:02:46.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.850+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 msgr2=0x7f16f813f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:46.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:46.850+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 0x7f16f813f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:02:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:47.651+0000 7f16f6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16f80b2210 legacy=0x7f16f8145ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-12-01T17:02:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:47.651+0000 7f16f77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 msgr2=0x7f16f80a8d70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-12-01T17:02:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:47.651+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 0x7f16f80a8d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:02:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:47.651+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 msgr2=0x7f16f813f620 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:47.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:47.651+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 0x7f16f813f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:02:49.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient(hunting): tick 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient(hunting): _check_auth_tickets 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 monclient(hunting): continuing hunt 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 msgr2=0x7f16f813f620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16f813f1b0 0x7f16f813f620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 msgr2=0x7f16f80a8d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16f80a8990 0x7f16f80a8d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16f80b2210 legacy=0x7f16f8145ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient: _add_conns ranks=[2,1] 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f16e4003690 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:02:49.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16e4006120 addr v1:172.21.15.92:6789/0 2023-12-01T17:02:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80aab40 con 0x7f16e4006120 2023-12-01T17:02:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:02:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8145640 con 0x7f16e4003690 2023-12-01T17:02:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f57fa640 10 monclient(hunting): _renew_subs 2023-12-01T17:02:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f77fe640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f77fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:02:49.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.248+0000 7f16f6ffd640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4006120 legacy=0x7f16e40085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.449+0000 7f16f77fe640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.449+0000 7f16f77fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:02:49.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.449+0000 7f16f6ffd640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4006120 legacy=0x7f16e40085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:49.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.850+0000 7f16f77fe640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:49.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.850+0000 7f16f77fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:02:49.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:49.850+0000 7f16f6ffd640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4006120 legacy=0x7f16e40085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:50.651+0000 7f16f77fe640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:50.651+0000 7f16f77fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:02:50.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:50.651+0000 7f16f6ffd640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4006120 legacy=0x7f16e40085e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): tick 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): _check_auth_tickets 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 monclient(hunting): continuing hunt 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4006120 legacy=0x7f16e40085e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f7fff640 1 -- reap_dead start 2023-12-01T17:02:52.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16e4009e70 addr v1:172.21.15.92:6789/0 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f16e4003690 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 0x7f16e4008560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f16e4008d10 addr v2:172.21.15.92:3300/0 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80aa910 con 0x7f16e4009e70 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f8145640 con 0x7f16e4008d10 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f81461e0 con 0x7f16e4003690 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f57fa640 10 monclient(hunting): _renew_subs 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f6ffd640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f77fe640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4009e70 legacy=0x7f16e400a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:52.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.248+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:02:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.449+0000 7f16f6ffd640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.449+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:02:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.449+0000 7f16f77fe640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4009e70 legacy=0x7f16e400a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.449+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.449+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:02:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.850+0000 7f16f6ffd640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.850+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:02:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.850+0000 7f16f77fe640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4009e70 legacy=0x7f16e400a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.850+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:52.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:52.850+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:02:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:53.651+0000 7f16f6ffd640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3301/0 2023-12-01T17:02:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:53.651+0000 7f16f6ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:02:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:53.651+0000 7f16f77fe640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4009e70 legacy=0x7f16e400a270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.92:6789/0 2023-12-01T17:02:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:53.651+0000 7f16f7fff640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.92:3300/0 2023-12-01T17:02:53.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:53.651+0000 7f16f7fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:02:53.705 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:00:07.464 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:02:54.020 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:00:07.780 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:02:54.356 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:00:08.137 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:02:54.356 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:00:08.137 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:02:54.690 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:00:08.471 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:02:54.690 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:00:08.471 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient(hunting): tick 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient(hunting): _check_auth_tickets 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 monclient(hunting): continuing hunt 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 -- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 msgr2=0x7f16e4005ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4003690 0x7f16e4005ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 -- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f16e4008d10 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 -- >> v1:172.21.15.92:6789/0 conn(0x7f16e4009e70 legacy=0x7f16e400a270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient: _add_conns ranks=[3,0] 2023-12-01T17:02:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f16e40091a0 addr v2:172.21.15.67:3300/0 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16e40100c0 addr v1:172.21.15.67:6789/0 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80aab40 con 0x7f16e40100c0 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f8145640 con 0x7f16e40091a0 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f57fa640 10 monclient(hunting): _renew_subs 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 msgr2=0x7f16e40095b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:02:55.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.248+0000 7f16f6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16e40100c0 legacy=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-12-01T17:02:55.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.449+0000 7f16f6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16e40100c0 legacy=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-12-01T17:02:55.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.449+0000 7f16f77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 msgr2=0x7f16e40095b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-12-01T17:02:55.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.449+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:02:55.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.850+0000 7f16f77fe640 1 -- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 msgr2=0x7f16e40095b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.67:3300/0 2023-12-01T17:02:55.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.850+0000 7f16f6ffd640 1 -- >> v1:172.21.15.67:6789/0 conn(0x7f16e40100c0 legacy=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.67:6789/0 2023-12-01T17:02:55.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:55.850+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:02:56.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.809+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:02:56.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.811+0000 7f16f77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36814/0 (socket says 172.21.15.67:36814) 2023-12-01T17:02:56.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.811+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 learned_addr learned my addr 172.21.15.67:0/3855957317 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:02:56.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.811+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e40091a0 auth_method 0 2023-12-01T17:02:56.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.811+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:02:56.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.811+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:02:56.812 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.811+0000 7f16f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:02:56.931 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.931+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 msgr2=0x7f16e40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:02:56.932 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.931+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 msgr2=0x7f16e40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:02:56.932 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.931+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:02:56.932 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:56.931+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:02:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): tick 2023-12-01T17:02:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): _check_auth_tickets 2023-12-01T17:02:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 monclient(hunting): continuing hunt 2023-12-01T17:02:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-01T17:02:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 msgr2=0x7f16e40095b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:02:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e40091a0 0x7f16e40095b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:02:58.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 >> v1:172.21.15.67:6789/0 conn(0x7f16e40100c0 legacy=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 reap_dead start 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16e4009e70 addr v1:172.21.15.92:6789/0 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16e40095b0 addr v1:172.21.15.67:6789/0 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f16e40051f0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80aa910 con 0x7f16e40095b0 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16e4005610 con 0x7f16e4009e70 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8145640 con 0x7f16e40051f0 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.248+0000 7f16f57fa640 10 monclient(hunting): _renew_subs 2023-12-01T17:02:58.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.249+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:02:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.249+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e40051f0 auth_method 0 2023-12-01T17:02:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.249+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:02:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.249+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:02:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.249+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:02:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.250+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-01T17:02:58.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.250+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:02:58.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.250+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:02:58.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.250+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:02:58.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.451+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:02:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.451+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e40051f0 auth_method 0 2023-12-01T17:02:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.452+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:02:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.452+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:02:58.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.452+0000 7f16f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:02:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.452+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-01T17:02:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.452+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:02:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.452+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:02:58.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.453+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:02:58.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.854+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.854+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e40051f0 auth_method 0 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.854+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.854+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.854+0000 7f16f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.855+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.855+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.855+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:02:58.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:58.855+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:02:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.656+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:02:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.656+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e40051f0 auth_method 0 2023-12-01T17:02:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.656+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:02:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.656+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:02:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.656+0000 7f16f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:02:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.658+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-01T17:02:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.658+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:02:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.658+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:02:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:02:59.658+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:03:01.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): tick 2023-12-01T17:03:01.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): _check_auth_tickets 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 monclient(hunting): continuing hunt 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40051f0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 >> v1:172.21.15.92:6789/0 conn(0x7f16e4009e70 legacy=0x7f16e400a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 >> v1:172.21.15.67:6789/0 conn(0x7f16e40095b0 legacy=0x7f16e4003ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient: _add_conns ranks=[3,4] 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f16e4005990 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f16e40129f0 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:01.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f8145640 con 0x7f16e4005990 2023-12-01T17:03:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f81461e0 con 0x7f16e40129f0 2023-12-01T17:03:01.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.249+0000 7f16f57fa640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.251+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.251+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e4005990 auth_method 0 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16e40129f0 auth_method 0 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:01.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:01.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:01.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:01.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:01.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.252+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:03:01.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.253+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:03:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.253+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.253+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:01.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.253+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:03:01.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.453+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:01.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.454+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e4005990 auth_method 0 2023-12-01T17:03:01.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.454+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:01.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.454+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:01.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.454+0000 7f16f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.454+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16e40129f0 auth_method 0 2023-12-01T17:03:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:01.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:01.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:01.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:03:01.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:03:01.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:01.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:01.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.455+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:03:01.856 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.856+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.856+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e4005990 auth_method 0 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.856+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.856+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.856+0000 7f16f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16e40129f0 auth_method 0 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:01.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:01.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:01.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:01.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:01.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.857+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:03:01.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.858+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:03:01.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.858+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:01.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.858+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:01.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:01.858+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:03:02.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.659+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:02.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.659+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:02.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.659+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e4005990 auth_method 0 2023-12-01T17:03:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.659+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.659+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.659+0000 7f16f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16e40129f0 auth_method 0 2023-12-01T17:03:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:02.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:02.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:02.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:03:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:03:02.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:02.660+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-12-01T17:03:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): tick 2023-12-01T17:03:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): _check_auth_tickets 2023-12-01T17:03:04.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 monclient(hunting): continuing hunt 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): _reopen_session rank -1 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e40129f0 0x7f16e4014de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 reap_dead start 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-c con 0x7f16e4005990 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f16e40095b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:04.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): picked mon.noname-a con 0x7f16e4009e70 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f8145640 con 0x7f16e4009e70 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f81461e0 con 0x7f16e4005990 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 1 -- 172.21.15.67:0/3855957317 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4009ac0 con 0x7f16e40095b0 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f57fa640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.249+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16e40095b0 auth_method 0 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e4005990 auth_method 0 2023-12-01T17:03:04.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:04.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:03:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.250+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:03:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.251+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.251+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.251+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.251+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:03:04.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.272+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.272+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e4009e70 auth_method 0 2023-12-01T17:03:04.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.272+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.272+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.272+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:04.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.275+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-12-01T17:03:04.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.275+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.275+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.275+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:03:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.452+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e4005990 auth_method 0 2023-12-01T17:03:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16e40095b0 auth_method 0 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.453+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.454+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.454+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.454+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.454+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:03:04.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.508+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.527+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e4009e70 auth_method 0 2023-12-01T17:03:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.527+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.527+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.527+0000 7f16f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.532+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.532+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 msgr2=0x7f16e4014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.532+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.532+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:03:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.854+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.854+0000 7f16f77fe640 10 monclient(hunting): get_auth_request con 0x7f16e4005990 auth_method 0 2023-12-01T17:03:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.854+0000 7f16f77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.854+0000 7f16f77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.854+0000 7f16f77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:04.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.855+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-12-01T17:03:04.856 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.855+0000 7f16f77fe640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.856 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.855+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.856 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.855+0000 7f16f77fe640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:03:04.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.858+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.858+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request con 0x7f16e40095b0 auth_method 0 2023-12-01T17:03:04.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.858+0000 7f16f6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.858+0000 7f16f6ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.858+0000 7f16f6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.886+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-12-01T17:03:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.886+0000 7f16f6ffd640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-12-01T17:03:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.886+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:03:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.886+0000 7f16f6ffd640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:03:04.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.948+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.67:3300/0 conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:04.957 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.957+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e4009e70 auth_method 0 2023-12-01T17:03:04.957 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.957+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:04.957 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.957+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:04.957 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.957+0000 7f16f7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:03:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.999+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.999+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:04.999+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.010+0000 7f16f7fff640 10 monclient(hunting): handle_auth_done global_id 4101 payload 274 2023-12-01T17:03:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.010+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 msgr2=0x7f16e4008560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:03:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.010+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.011+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 msgr2=0x7f16e40124b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-12-01T17:03:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.011+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.011+0000 7f16f7fff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.011+0000 7f16f7fff640 1 monclient: found mon.noname-a 2023-12-01T17:03:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.011+0000 7f16f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.011+0000 7f16f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:03:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.011+0000 7f16f7fff640 1 -- 172.21.15.67:0/3855957317 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e40057a0 con 0x7f16e4009e70 2023-12-01T17:03:05.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.028+0000 7f16f7fff640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16e4014de0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f16e400ad80 tx=0x7f16f002ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d934600c965d3d0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 1 -- 172.21.15.67:0/3855957317 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f002d850 con 0x7f16e4009e70 2023-12-01T17:03:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:03:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: dump: 2023-12-01T17:03:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: _finish_auth 0 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: _check_auth_tickets 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 1 -- 172.21.15.67:0/3855957317 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f0031070 con 0x7f16e4009e70 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16fdb9e640 5 monclient: authenticate success, global_id 4101 2023-12-01T17:03:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16fdb9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16fdb9e640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 1 -- 172.21.15.67:0/3855957317 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f002dbb0 con 0x7f16e4009e70 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16fdb9e640 4 set_mon_vals no callback set 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16f67fc640 10 monclient: dump: 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16fdb9e640 10 monclient: shutdown 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16fdb9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.098+0000 7f16fdb9e640 1 -- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 msgr2=0x7f16e4014de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16e4014de0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f16e400ad80 tx=0x7f16f002ceb0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 -- 172.21.15.67:0/3855957317 shutdown_connections 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16e4014de0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/3855957317 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e40095b0 0x7f16e4008560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/3855957317 >> v2:172.21.15.92:3300/0 conn(0x7f16e4005990 0x7f16e40124b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 -- 172.21.15.67:0/3855957317 >> 172.21.15.67:0/3855957317 conn(0x7f16f80b83c0 msgr2=0x7f16f80b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 -- 172.21.15.67:0/3855957317 shutdown_connections 2023-12-01T17:03:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 1 -- 172.21.15.67:0/3855957317 wait complete. 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 10 monclient: build_initial_monmap 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.099+0000 7f16fdb9e640 10 monclient: monmap: 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 1 librados: starting msgr at 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 1 librados: starting objecter 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 1 Processor -- start 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 1 -- start start 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 1 librados: setting wanted keys 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 1 librados: calling monclient init 2023-12-01T17:03:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 10 monclient: init 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.100+0000 7f16fdb9e640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4005990 0x7f16f8144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): picked mon.noname-c con 0x7f16e4005990 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16e40095b0 0x7f16f81418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): picked mon.noname-b con 0x7f16e40095b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16f8141e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): picked mon.noname-a con 0x7f16e4009e70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8140de0 con 0x7f16e4009e70 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f8144ff0 con 0x7f16e40095b0 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f813fbd0 con 0x7f16e4005990 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:05.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.101+0000 7f16fdb9e640 10 monclient(hunting): authenticate will time out at 1316.338464s 2023-12-01T17:03:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.107+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16f8141e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.108+0000 7f16f7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16f8141e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39940/0 (socket says 172.21.15.67:39940) 2023-12-01T17:03:05.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.108+0000 7f16f7fff640 1 -- 172.21.15.67:0/1176823278 learned_addr learned my addr 172.21.15.67:0/1176823278 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.108+0000 7f16f7fff640 10 monclient(hunting): get_auth_request con 0x7f16e4009e70 auth_method 0 2023-12-01T17:03:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.108+0000 7f16f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.108+0000 7f16f7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.108+0000 7f16f7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.109+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.109+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.109+0000 7f16f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 10 monclient(hunting): handle_auth_done global_id 4107 payload 824 2023-12-01T17:03:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 1 -- 172.21.15.67:0/1176823278 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4005990 msgr2=0x7f16f8144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 1 --2- 172.21.15.67:0/1176823278 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4005990 0x7f16f8144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 1 -- 172.21.15.67:0/1176823278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16e40095b0 msgr2=0x7f16f81418c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 1 --2- 172.21.15.67:0/1176823278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16e40095b0 0x7f16f81418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 1 monclient: found mon.noname-a 2023-12-01T17:03:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:03:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.119+0000 7f16f7fff640 1 -- 172.21.15.67:0/1176823278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f000b040 con 0x7f16e4009e70 2023-12-01T17:03:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.124+0000 7f16f7fff640 1 --2- 172.21.15.67:0/1176823278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16f8141e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16f0039040 tx=0x7f16f0003ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.124+0000 7f16dffff640 10 client.?.objecter ms_handle_connect 0x7f16e4009e70 2023-12-01T17:03:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.124+0000 7f16dffff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 1 -- 172.21.15.67:0/1176823278 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0047070 con 0x7f16e4009e70 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: dump: 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:05.163 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: _finish_auth 0 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: _check_auth_tickets 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 1 -- 172.21.15.67:0/1176823278 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f0009040 con 0x7f16e4009e70 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 1 -- 172.21.15.67:0/1176823278 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f0007160 con 0x7f16e4009e70 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 5 monclient: authenticate success, global_id 4107 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 10 monclient: _renew_subs 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f8143fa0 con 0x7f16e4009e70 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16f5ffb640 4 set_mon_vals no callback set 2023-12-01T17:03:05.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16dffff640 10 monclient: dump: 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 10 monclient: _renew_subs 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:03:05.165 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f8142520 con 0x7f16e4009e70 2023-12-01T17:03:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.162+0000 7f16fdb9e640 1 librados: init done 2023-12-01T17:03:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.163+0000 7f16fdb9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:05.166 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.163+0000 7f16fdb9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.163+0000 7f16fdb9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:03:05.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.163+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f80aa070 con 0x7f16e4009e70 2023-12-01T17:03:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.183+0000 7f16dffff640 1 -- 172.21.15.67:0/1176823278 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f16f003d040 con 0x7f16e4009e70 2023-12-01T17:03:05.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.183+0000 7f16dffff640 1 -- 172.21.15.67:0/1176823278 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f16f00415d0 con 0x7f16e4009e70 2023-12-01T17:03:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.183+0000 7f16dffff640 10 client.4107.objecter ms_dispatch 0x7f16f8000b90 osd_map(1..1 src has 1..1) v4 2023-12-01T17:03:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.183+0000 7f16dffff640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2023-12-01T17:03:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.183+0000 7f16dffff640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2023-12-01T17:03:05.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.183+0000 7f16dffff640 20 client.4107.objecter dump_active .. 0 homeless 2023-12-01T17:03:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.358+0000 7f16dffff640 1 -- 172.21.15.67:0/1176823278 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f16f003b070 con 0x7f16e4009e70 2023-12-01T17:03:05.358 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.358+0000 7f16dffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:05.359 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.358+0000 7f16dffff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.426+0000 7f16fdb9e640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-01T17:03:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.426+0000 7f16fdb9e640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-01T17:03:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.426+0000 7f16fdb9e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:03:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.426+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f16f8141300 con 0x7f16e4009e70 2023-12-01T17:03:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.539+0000 7f16dffff640 1 -- 172.21.15.67:0/1176823278 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f16f8141300 con 0x7f16e4009e70 2023-12-01T17:03:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.539+0000 7f16dffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-12-01T17:03:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.539+0000 7f16dffff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-12-01T17:03:05.540 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2023-12-01T17:03:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 10 librados: watch_flush enter 2023-12-01T17:03:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 10 librados: watch_flush exit 2023-12-01T17:03:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 20 client.4107.objecter shutdown clearing up homeless session... 2023-12-01T17:03:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 10 client.4107.objecter successfully canceled tick 2023-12-01T17:03:05.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 10 monclient: shutdown 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 msgr2=0x7f16f8141e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/1176823278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16f8141e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f16f0039040 tx=0x7f16f0003ec0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 shutdown_connections 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/1176823278 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f16e4009e70 0x7f16f8141e00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/1176823278 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f16e40095b0 0x7f16f81418c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 --2- 172.21.15.67:0/1176823278 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f16e4005990 0x7f16f8144ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 >> 172.21.15.67:0/1176823278 conn(0x7f16f80b83c0 msgr2=0x7f16f80b5400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 shutdown_connections 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 -- 172.21.15.67:0/1176823278 wait complete. 2023-12-01T17:03:05.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:05.540+0000 7f16fdb9e640 1 librados: shutdown 2023-12-01T17:03:05.553 INFO:tasks.ceph:check_enable_crimson: False 2023-12-01T17:03:05.553 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-12-01T17:03:05.553 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:03:05.554 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-12-01T17:03:05.586 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:03:05.586 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-12-01T17:03:05.657 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:03:05.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-12-01T17:03:05.725 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:05.724+0000 7fabd1a2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-01T17:03:05.728 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:03:05.728 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-12-01T17:03:05.784 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:05.783+0000 7fabd1a2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-01T17:03:05.800 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:03:05.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-12-01T17:03:05.810 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:05.811+0000 7f4645e3d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-01T17:03:05.828 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:03:05.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-12-01T17:03:05.872 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:05.872+0000 7f4645e3d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-01T17:03:05.894 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:03:05.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-12-01T17:03:05.964 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-12-01T17:03:05.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-12-01T17:03:06.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new fbc446fc-b047-4553-83da-1ab39870ff2c 0 2023-12-01T17:03:06.171 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.171+0000 7faa09e7c640 10 monclient: get_monmap_and_config 2023-12-01T17:03:06.171 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.171+0000 7faa09e7c640 10 monclient: build_initial_monmap 2023-12-01T17:03:06.171 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.171+0000 7faa09e7c640 10 monclient: monmap: 2023-12-01T17:03:06.171 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:06.172143+0000 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:06.172143+0000 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:06.172 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.173 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.172+0000 7faa09e7c640 1 Processor -- start 2023-12-01T17:03:06.173 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.172+0000 7faa09e7c640 1 -- start start 2023-12-01T17:03:06.173 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.172+0000 7faa09e7c640 10 monclient: init 2023-12-01T17:03:06.173 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa0406f550 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa041421b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient(hunting): picked mon.noname-b con 0x7faa04071f10 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa041426f0 0x7faa04149390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient(hunting): picked mon.noname-a con 0x7faa041426f0 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0406fbe0 con 0x7faa0406f550 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0406f930 con 0x7faa041426f0 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa04071810 con 0x7faa04071f10 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa09e7c640 10 monclient(hunting): authenticate will time out at 1329.483971s 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa037fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7faa0406f550 0x7faa0413bef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:41234/0 (socket says 172.21.15.92:41234) 2023-12-01T17:03:06.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.173+0000 7faa037fe640 1 -- 172.21.15.92:0/2516531786 learned_addr learned my addr 172.21.15.92:0/2516531786 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.174+0000 7fa9faffd640 1 --2- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa041421b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.174+0000 7faa03fff640 1 --2- 172.21.15.92:0/2516531786 >> v2:172.21.15.67:3300/0 conn(0x7faa041426f0 0x7faa04149390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.174+0000 7fa9faffd640 10 monclient(hunting): get_auth_request con 0x7faa04071f10 auth_method 0 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7fa9faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7fa9faffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7fa9faffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7faa03fff640 10 monclient(hunting): get_auth_request con 0x7faa041426f0 auth_method 0 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7faa03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7faa03fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7faa03fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7fa9faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7fa9faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.177 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.175+0000 7fa9faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.177 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.176+0000 7faa02ffd640 1 -- 172.21.15.92:0/2516531786 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231589849 0 0) 0x7faa0406fbe0 con 0x7faa0406f550 2023-12-01T17:03:06.177 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.176+0000 7faa02ffd640 10 monclient(hunting): _negotiate 2023-12-01T17:03:06.177 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.176+0000 7faa02ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.176+0000 7faa02ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.176+0000 7faa02ffd640 10 monclient(hunting): my global_id is 4108 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.176+0000 7faa02ffd640 1 -- 172.21.15.92:0/2516531786 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9f0003520 con 0x7faa0406f550 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 10 monclient(hunting): handle_auth_done global_id 4110 payload 274 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 1 -- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 msgr2=0x7faa041421b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 1 --2- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa041421b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 1 -- 172.21.15.92:0/2516531786 >> v1:172.21.15.92:6789/0 conn(0x7faa0406f550 legacy=0x7faa0413bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 1 monclient: found mon.noname-a 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:03:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.177+0000 7faa03fff640 1 -- 172.21.15.92:0/2516531786 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa04071af0 con 0x7faa041426f0 2023-12-01T17:03:06.179 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.179+0000 7faa03fff640 1 --2- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa041426f0 0x7faa04149390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009d90 tx=0x7fa9fc02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dbf4fbeff1d30f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 1 -- 172.21.15.92:0/2516531786 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc02cc50 con 0x7faa041426f0 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: dump: 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.183 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: _finish_auth 0 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 1 -- 172.21.15.92:0/2516531786 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc02cdb0 con 0x7faa041426f0 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 1 -- 172.21.15.92:0/2516531786 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc034660 con 0x7faa041426f0 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa09e7c640 5 monclient: authenticate success, global_id 4110 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa09e7c640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa09e7c640 4 set_mon_vals no callback set 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.183+0000 7faa02ffd640 10 monclient: dump: 2023-12-01T17:03:06.184 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.185 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 10 monclient: shutdown 2023-12-01T17:03:06.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:06.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 -- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa041426f0 msgr2=0x7faa04149390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 --2- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa041426f0 0x7faa04149390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009d90 tx=0x7fa9fc02bea0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 -- 172.21.15.92:0/2516531786 shutdown_connections 2023-12-01T17:03:06.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 --2- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa041426f0 0x7faa04149390 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 --2- 172.21.15.92:0/2516531786 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa041421b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 -- 172.21.15.92:0/2516531786 >> 172.21.15.92:0/2516531786 conn(0x7faa0406c100 msgr2=0x7faa040b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 -- 172.21.15.92:0/2516531786 shutdown_connections 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 1 -- 172.21.15.92:0/2516531786 wait complete. 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 10 monclient: build_initial_monmap 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.184+0000 7faa09e7c640 10 monclient: monmap: 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:06.189 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 1 librados: starting msgr at 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 1 librados: starting objecter 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 1 Processor -- start 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 1 -- start start 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 1 librados: setting wanted keys 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 1 librados: calling monclient init 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 10 monclient: init 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.185+0000 7faa09e7c640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa0406f550 0x7faa041d60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): picked mon.noname-a con 0x7faa0406f550 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa04141340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.190 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): picked mon.noname-c con 0x7faa04071f10 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa041426f0 0x7faa04141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): picked mon.noname-b con 0x7faa041426f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0413ec60 con 0x7faa0406f550 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa041380c0 con 0x7faa041426f0 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa04141dc0 con 0x7faa04071f10 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa09e7c640 10 monclient(hunting): authenticate will time out at 1329.496799s 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa03fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa041426f0 0x7faa04141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa037fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa0406f550 0x7faa041d60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7fa9faffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa04141340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa03fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa041426f0 0x7faa04141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43902/0 (socket says 172.21.15.92:43902) 2023-12-01T17:03:06.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa03fff640 1 -- 172.21.15.92:0/898763117 learned_addr learned my addr 172.21.15.92:0/898763117 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa03fff640 10 monclient(hunting): get_auth_request con 0x7faa041426f0 auth_method 0 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa03fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.186+0000 7faa03fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7fa9faffd640 10 monclient(hunting): get_auth_request con 0x7faa04071f10 auth_method 0 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7fa9faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7fa9faffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7fa9faffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa037fe640 10 monclient(hunting): get_auth_request con 0x7faa0406f550 auth_method 0 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa037fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa037fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7faa037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7fa9faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7fa9faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.187+0000 7fa9faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 1 -- 172.21.15.92:0/898763117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 msgr2=0x7faa04141340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 1 --2- 172.21.15.92:0/898763117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa04141340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 1 -- 172.21.15.92:0/898763117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa0406f550 msgr2=0x7faa041d60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 1 --2- 172.21.15.92:0/898763117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa0406f550 0x7faa041d60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 1 monclient: found mon.noname-b 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.190+0000 7faa03fff640 1 -- 172.21.15.92:0/898763117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc009a70 con 0x7faa041426f0 2023-12-01T17:03:06.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.192+0000 7faa03fff640 1 --2- 172.21.15.92:0/898763117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa041426f0 0x7faa04141880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009eb0 tx=0x7fa9fc004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:06.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.192+0000 7faa00ff9640 10 client.?.objecter ms_handle_connect 0x7faa041426f0 2023-12-01T17:03:06.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.192+0000 7faa00ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:06.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 1 -- 172.21.15.92:0/898763117 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc034e40 con 0x7faa041426f0 2023-12-01T17:03:06.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:06.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: dump: 2023-12-01T17:03:06.196 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.196 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.196 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: _finish_auth 0 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 1 -- 172.21.15.92:0/898763117 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc03f420 con 0x7faa041426f0 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 1 -- 172.21.15.92:0/898763117 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc03f7b0 con 0x7faa041426f0 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 5 monclient: authenticate success, global_id 4111 2023-12-01T17:03:06.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 10 monclient: _renew_subs 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa041d0400 con 0x7faa041426f0 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa027fc640 4 set_mon_vals no callback set 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa00ff9640 10 monclient: dump: 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.198 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.199 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:06.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 10 monclient: _renew_subs 2023-12-01T17:03:06.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa04139190 con 0x7faa041426f0 2023-12-01T17:03:06.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.196+0000 7faa09e7c640 1 librados: init done 2023-12-01T17:03:06.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.197+0000 7faa09e7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:06.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.197+0000 7faa09e7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:06.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.197+0000 7faa09e7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.197+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0404e840 con 0x7faa041426f0 2023-12-01T17:03:06.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.200+0000 7faa00ff9640 1 -- 172.21.15.92:0/898763117 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa9fc03fc10 con 0x7faa041426f0 2023-12-01T17:03:06.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.200+0000 7faa00ff9640 1 -- 172.21.15.92:0/898763117 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa9fc052be0 con 0x7faa041426f0 2023-12-01T17:03:06.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.200+0000 7faa00ff9640 10 client.4111.objecter ms_dispatch 0x7faa04000b90 osd_map(2..2 src has 1..2) v4 2023-12-01T17:03:06.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.200+0000 7faa00ff9640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2023-12-01T17:03:06.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.200+0000 7faa00ff9640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2023-12-01T17:03:06.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.200+0000 7faa00ff9640 20 client.4111.objecter dump_active .. 0 homeless 2023-12-01T17:03:06.349 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.349+0000 7fabd1a2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-01T17:03:06.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.372+0000 7faa00ff9640 1 -- 172.21.15.92:0/898763117 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fa9fc04d060 con 0x7faa041426f0 2023-12-01T17:03:06.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.372+0000 7faa00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:06.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.372+0000 7faa00ff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:06.404 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.403+0000 7fabd1a2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-01T17:03:06.433 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.433+0000 7f4645e3d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-01T17:03:06.445 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.445+0000 7faa09e7c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "fbc446fc-b047-4553-83da-1ab39870ff2c", "id": 0}] 2023-12-01T17:03:06.445 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.445+0000 7faa09e7c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "fbc446fc-b047-4553-83da-1ab39870ff2c", "id": 0}] 2023-12-01T17:03:06.445 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.445+0000 7faa09e7c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.445 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.445+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fbc446fc-b047-4553-83da-1ab39870ff2c", "id": 0} v 0) v1 -- 0x7faa040aff50 con 0x7faa041426f0 2023-12-01T17:03:06.450 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.450+0000 7fabd1a2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-01T17:03:06.488 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.488+0000 7f4645e3d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-01T17:03:06.518 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.518+0000 7fabd1a2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-01T17:03:06.532 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.532+0000 7f4645e3d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-01T17:03:06.565 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.565+0000 7fabd1a2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-01T17:03:06.586 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.586+0000 7f4645e3d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-01T17:03:06.629 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.629+0000 7f4645e3d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-01T17:03:06.661 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.661+0000 7faa00ff9640 1 -- 172.21.15.92:0/898763117 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fbc446fc-b047-4553-83da-1ab39870ff2c", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa9fc0570b0 con 0x7faa041426f0 2023-12-01T17:03:06.662 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.661+0000 7faa00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "fbc446fc-b047-4553-83da-1ab39870ff2c", "id": 0}] 2023-12-01T17:03:06.662 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.661+0000 7faa00ff9640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:06.662 INFO:teuthology.orchestra.run.smithi092.stdout:0 2023-12-01T17:03:06.665 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.661+0000 7faa09e7c640 10 librados: watch_flush enter 2023-12-01T17:03:06.665 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.661+0000 7faa09e7c640 10 librados: watch_flush exit 2023-12-01T17:03:06.665 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.661+0000 7faa09e7c640 20 client.4111.objecter shutdown clearing up homeless session... 2023-12-01T17:03:06.665 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 10 client.4111.objecter successfully canceled tick 2023-12-01T17:03:06.665 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 10 monclient: shutdown 2023-12-01T17:03:06.665 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:06.665 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa041426f0 msgr2=0x7faa04141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 --2- 172.21.15.92:0/898763117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa041426f0 0x7faa04141880 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009eb0 tx=0x7fa9fc004a40 comp rx=0 tx=0).stop 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 shutdown_connections 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 --2- 172.21.15.92:0/898763117 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa041426f0 0x7faa04141880 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 --2- 172.21.15.92:0/898763117 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa04071f10 0x7faa04141340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 --2- 172.21.15.92:0/898763117 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa0406f550 0x7faa041d60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 >> 172.21.15.92:0/898763117 conn(0x7faa0406c100 msgr2=0x7faa0413bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 shutdown_connections 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 -- 172.21.15.92:0/898763117 wait complete. 2023-12-01T17:03:06.666 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.662+0000 7faa09e7c640 1 librados: shutdown 2023-12-01T17:03:06.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 333c878f-1386-495c-a6d6-1fd5e68c005e 1 2023-12-01T17:03:06.679 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.679+0000 7fabd1a2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-01T17:03:06.739 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.740+0000 7f4645e3d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-12-01T17:03:06.754 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.754+0000 7fabd1a2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-01T17:03:06.773 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.773+0000 7fec7c361640 10 monclient: get_monmap_and_config 2023-12-01T17:03:06.773 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.773+0000 7fec7c361640 10 monclient: build_initial_monmap 2023-12-01T17:03:06.773 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.773+0000 7fec7c361640 10 monclient: monmap: 2023-12-01T17:03:06.773 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:06.773 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:06.773518+0000 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:06.773518+0000 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.773+0000 7fec7c361640 1 Processor -- start 2023-12-01T17:03:06.775 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.773+0000 7fec7c361640 1 -- start start 2023-12-01T17:03:06.775 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.773+0000 7fec7c361640 10 monclient: init 2023-12-01T17:03:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:06.776 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec740a8930 0x7fec740a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient(hunting): picked mon.noname-c con 0x7fec740a8930 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec740b21a0 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec7413f0b0 0x7fec7413f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient(hunting): picked mon.noname-b con 0x7fec7413f0b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec740aa980 con 0x7fec740b21a0 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7413fa20 con 0x7fec740a8930 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec740aabf0 con 0x7fec7413f0b0 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.774+0000 7fec7c361640 10 monclient(hunting): authenticate will time out at 1330.085277s 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fec740a8930 0x7fec740a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec7413f0b0 0x7fec7413f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec798fc640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fec740b21a0 0x7fec741424b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.92:57020/0 (socket says 172.21.15.92:57020) 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec798fc640 1 -- 172.21.15.92:0/2382202074 learned_addr learned my addr 172.21.15.92:0/2382202074 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 10 monclient(hunting): get_auth_request con 0x7fec7413f0b0 auth_method 0 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 10 monclient(hunting): get_auth_request con 0x7fec740a8930 auth_method 0 2023-12-01T17:03:06.778 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a0fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.779 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.775+0000 7fec7a8fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.780 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2023-12-01T17:03:06.780 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 1 -- 172.21.15.92:0/2382202074 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec7413f0b0 msgr2=0x7fec7413f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:06.780 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 1 --2- 172.21.15.92:0/2382202074 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec7413f0b0 0x7fec7413f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.780 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 1 -- 172.21.15.92:0/2382202074 >> v1:172.21.15.67:6789/0 conn(0x7fec740b21a0 legacy=0x7fec741424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.781 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 10 monclient: _finish_hunting 0 2023-12-01T17:03:06.781 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 1 monclient: found mon.noname-c 2023-12-01T17:03:06.781 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:06.781 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.781 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.777+0000 7fec7a0fd640 1 -- 172.21.15.92:0/2382202074 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec740aaed0 con 0x7fec740a8930 2023-12-01T17:03:06.782 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.779+0000 7fec7a0fd640 1 --2- 172.21.15.92:0/2382202074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740a8930 0x7fec740a8d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fec640097d0 tx=0x7fec6402b810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=535724d2af09b86e server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:06.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 1 -- 172.21.15.92:0/2382202074 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64031770 con 0x7fec740a8930 2023-12-01T17:03:06.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.787 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:06.788 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: dump: 2023-12-01T17:03:06.788 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.788 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.788 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.788 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.788 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.789 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.789 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.789 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.789 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.789 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.789 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: _finish_auth 0 2023-12-01T17:03:06.789 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: _check_auth_tickets 2023-12-01T17:03:06.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:06.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 1 -- 172.21.15.92:0/2382202074 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec640318d0 con 0x7fec740a8930 2023-12-01T17:03:06.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec7c361640 5 monclient: authenticate success, global_id 4114 2023-12-01T17:03:06.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec7c361640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:06.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:06.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec7c361640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:06.790 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec7c361640 4 set_mon_vals no callback set 2023-12-01T17:03:06.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 1 -- 172.21.15.92:0/2382202074 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec64031c60 con 0x7fec740a8930 2023-12-01T17:03:06.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:06.791 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.782+0000 7fec790fb640 10 monclient: dump: 2023-12-01T17:03:06.791 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.792 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 10 monclient: shutdown 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 -- 172.21.15.92:0/2382202074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740a8930 msgr2=0x7fec740a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 --2- 172.21.15.92:0/2382202074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740a8930 0x7fec740a8d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fec640097d0 tx=0x7fec6402b810 comp rx=0 tx=0).stop 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 -- 172.21.15.92:0/2382202074 shutdown_connections 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 --2- 172.21.15.92:0/2382202074 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec7413f0b0 0x7fec7413f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 --2- 172.21.15.92:0/2382202074 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740a8930 0x7fec740a8d10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.793 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 -- 172.21.15.92:0/2382202074 >> 172.21.15.92:0/2382202074 conn(0x7fec740b82f0 msgr2=0x7fec740b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 -- 172.21.15.92:0/2382202074 shutdown_connections 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.783+0000 7fec7c361640 1 -- 172.21.15.92:0/2382202074 wait complete. 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 10 monclient: build_initial_monmap 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 10 monclient: monmap: 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:06.795 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 1 librados: starting msgr at 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 1 librados: starting objecter 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 1 Processor -- start 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 1 -- start start 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 1 librados: setting wanted keys 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 1 librados: calling monclient init 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.784+0000 7fec7c361640 10 monclient: init 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:03:06.796 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec740a8930 0x7fec740b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): picked mon.noname-c con 0x7fec740a8930 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740b21a0 0x7fec74145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): picked mon.noname-b con 0x7fec740b21a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec7413f0b0 0x7fec74143150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): picked mon.noname-a con 0x7fec7413f0b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec741387f0 con 0x7fec7413f0b0 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec74145ac0 con 0x7fec740b21a0 2023-12-01T17:03:06.797 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec74145fe0 con 0x7fec740a8930 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec7c361640 10 monclient(hunting): authenticate will time out at 1330.096259s 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.785+0000 7fec798fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740b21a0 0x7fec74145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec740a8930 0x7fec740b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec740a8930 0x7fec740b68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.92:54506/0 (socket says 172.21.15.92:54506) 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 1 -- 172.21.15.92:0/2940354220 learned_addr learned my addr 172.21.15.92:0/2940354220 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 10 monclient(hunting): get_auth_request con 0x7fec740a8930 auth_method 0 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a8fe640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec7413f0b0 0x7fec74143150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:06.798 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec798fc640 10 monclient(hunting): get_auth_request con 0x7fec740b21a0 auth_method 0 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec798fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec798fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec798fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a8fe640 10 monclient(hunting): get_auth_request con 0x7fec7413f0b0 auth_method 0 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a8fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a8fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a8fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec7a0fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec798fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec798fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.786+0000 7fec798fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.787+0000 7fec7a8fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:06.799 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.787+0000 7fec7a8fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:06.800 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.787+0000 7fec7a8fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:06.801 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2023-12-01T17:03:06.801 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 1 -- 172.21.15.92:0/2940354220 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec740a8930 msgr2=0x7fec740b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.801 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec740a8930 0x7fec740b68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.801 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 1 -- 172.21.15.92:0/2940354220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec7413f0b0 msgr2=0x7fec74143150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec7413f0b0 0x7fec74143150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 10 monclient: _finish_hunting 0 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 1 monclient: found mon.noname-b 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.789+0000 7fec798fc640 1 -- 172.21.15.92:0/2940354220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec64009400 con 0x7fec740b21a0 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.791+0000 7fec798fc640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740b21a0 0x7fec74145090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fec70000f60 tx=0x7fec7000c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.791+0000 7fec6affd640 10 client.?.objecter ms_handle_connect 0x7fec740b21a0 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.791+0000 7fec6affd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 1 -- 172.21.15.92:0/2940354220 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70005ce0 con 0x7fec740b21a0 2023-12-01T17:03:06.802 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: dump: 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: _finish_auth 0 2023-12-01T17:03:06.803 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: _check_auth_tickets 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 5 monclient: authenticate success, global_id 4117 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 1 -- 172.21.15.92:0/2940354220 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70005e40 con 0x7fec740b21a0 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 1 -- 172.21.15.92:0/2940354220 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70007310 con 0x7fec740b21a0 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec6affd640 10 monclient: dump: 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:06.804 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 10 monclient: _renew_subs 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec788fa640 4 set_mon_vals no callback set 2023-12-01T17:03:06.805 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.806 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec741455d0 con 0x7fec740b21a0 2023-12-01T17:03:06.806 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:06.806 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 10 monclient: _renew_subs 2023-12-01T17:03:06.806 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.806 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec74143870 con 0x7fec740b21a0 2023-12-01T17:03:06.806 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.795+0000 7fec7c361640 1 librados: init done 2023-12-01T17:03:06.807 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.796+0000 7fec7c361640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:06.807 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.796+0000 7fec7c361640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:06.807 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.796+0000 7fec7c361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.807 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.796+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec7404e840 con 0x7fec740b21a0 2023-12-01T17:03:06.808 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.803+0000 7fabd1a2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-01T17:03:06.814 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.814+0000 7f4645e3d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-01T17:03:06.858 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.858+0000 7fabd1a2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-01T17:03:06.861 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.862+0000 7f4645e3d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-01T17:03:06.898 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 1 -- 172.21.15.92:0/2940354220 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fec7001c440 con 0x7fec740b21a0 2023-12-01T17:03:06.898 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 1 -- 172.21.15.92:0/2940354220 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7fec700127b0 con 0x7fec740b21a0 2023-12-01T17:03:06.898 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 10 client.4117.objecter ms_dispatch 0x7fec74000b90 osd_map(3..3 src has 1..3) v4 2023-12-01T17:03:06.898 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2023-12-01T17:03:06.898 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2023-12-01T17:03:06.898 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 20 client.4117.objecter dump_active .. 0 homeless 2023-12-01T17:03:06.899 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 1 -- 172.21.15.92:0/2940354220 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fec70017070 con 0x7fec740b21a0 2023-12-01T17:03:06.899 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:06.899 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.898+0000 7fec6affd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:06.907 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:06.907+0000 7fabd1a2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-01T17:03:06.916 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.916+0000 7f4645e3d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-01T17:03:06.967 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:06.964+0000 7f4645e3d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-01T17:03:06.970 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.970+0000 7fec7c361640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "333c878f-1386-495c-a6d6-1fd5e68c005e", "id": 1}] 2023-12-01T17:03:06.970 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.970+0000 7fec7c361640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "333c878f-1386-495c-a6d6-1fd5e68c005e", "id": 1}] 2023-12-01T17:03:06.970 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.970+0000 7fec7c361640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:06.970 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:06.970+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "333c878f-1386-495c-a6d6-1fd5e68c005e", "id": 1} v 0) v1 -- 0x7fec740b7540 con 0x7fec740b21a0 2023-12-01T17:03:07.061 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.061+0000 7fec6affd640 1 -- 172.21.15.92:0/2940354220 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "333c878f-1386-495c-a6d6-1fd5e68c005e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fec70012ae0 con 0x7fec740b21a0 2023-12-01T17:03:07.061 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.061+0000 7fec6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "333c878f-1386-495c-a6d6-1fd5e68c005e", "id": 1}] 2023-12-01T17:03:07.061 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.061+0000 7fec6affd640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:07.062 INFO:teuthology.orchestra.run.smithi092.stdout:1 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 10 librados: watch_flush enter 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 10 librados: watch_flush exit 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 20 client.4117.objecter shutdown clearing up homeless session... 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 10 client.4117.objecter successfully canceled tick 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 10 monclient: shutdown 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740b21a0 msgr2=0x7fec74145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740b21a0 0x7fec74145090 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fec70000f60 tx=0x7fec7000c800 comp rx=0 tx=0).stop 2023-12-01T17:03:07.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 shutdown_connections 2023-12-01T17:03:07.064 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fec7413f0b0 0x7fec74143150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.064 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fec740b21a0 0x7fec74145090 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.064 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 --2- 172.21.15.92:0/2940354220 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fec740a8930 0x7fec740b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.064 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 >> 172.21.15.92:0/2940354220 conn(0x7fec740b82f0 msgr2=0x7fec74137340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:07.064 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 shutdown_connections 2023-12-01T17:03:07.064 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 -- 172.21.15.92:0/2940354220 wait complete. 2023-12-01T17:03:07.064 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.062+0000 7fec7c361640 1 librados: shutdown 2023-12-01T17:03:07.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new f72d775b-8e17-4768-87e3-6cc7a2bdb6a3 2 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.171+0000 7f968cb0f640 10 monclient: get_monmap_and_config 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.171+0000 7f968cb0f640 10 monclient: build_initial_monmap 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.171+0000 7f968cb0f640 10 monclient: monmap: 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:07.172083+0000 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:07.172083+0000 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:07.172 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:07.173 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:07.173 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:07.173 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:07.173 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.172+0000 7f968cb0f640 1 Processor -- start 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.172+0000 7f968cb0f640 1 -- start start 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.172+0000 7f968cb0f640 10 monclient: init 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f96880a8930 0x7f96880a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): picked mon.noname-c con 0x7f96880a8930 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96880b21a0 0x7f96881424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): picked mon.noname-a con 0x7f96880b21a0 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96881429f0 0x7f968813f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.174 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): picked mon.noname-b con 0x7f96881429f0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96880aa980 con 0x7f96880b21a0 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96880aabf0 con 0x7f96880a8930 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96880aaed0 con 0x7f96881429f0 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:07.175 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.173+0000 7f968cb0f640 10 monclient(hunting): authenticate will time out at 1330.483867s 2023-12-01T17:03:07.180 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.180+0000 7f9686d9d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96881429f0 0x7f968813f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.181+0000 7f9685d9b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f96880b21a0 0x7f96881424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96881429f0 0x7f968813f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.92:54508/0 (socket says 172.21.15.92:54508) 2023-12-01T17:03:07.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 1 -- 172.21.15.92:0/4103326831 learned_addr learned my addr 172.21.15.92:0/4103326831 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:07.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 10 monclient(hunting): get_auth_request con 0x7f96881429f0 auth_method 0 2023-12-01T17:03:07.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9685d9b640 10 monclient(hunting): get_auth_request con 0x7f96880b21a0 auth_method 0 2023-12-01T17:03:07.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9685d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.182 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9685d9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9685d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.182+0000 7f9686d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.183+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.183+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.183 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.183+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2023-12-01T17:03:07.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 1 -- 172.21.15.92:0/4103326831 >> v2:172.21.15.92:3300/0 conn(0x7f96880a8930 msgr2=0x7f96880a8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 1 --2- 172.21.15.92:0/4103326831 >> v2:172.21.15.92:3300/0 conn(0x7f96880a8930 0x7f96880a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 1 -- 172.21.15.92:0/4103326831 >> v2:172.21.15.67:3300/0 conn(0x7f96880b21a0 msgr2=0x7f96881424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 1 --2- 172.21.15.92:0/4103326831 >> v2:172.21.15.67:3300/0 conn(0x7f96880b21a0 0x7f96881424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 1 monclient: found mon.noname-b 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:03:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.185+0000 7f9686d9d640 1 -- 172.21.15.92:0/4103326831 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9688141f90 con 0x7f96881429f0 2023-12-01T17:03:07.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.188+0000 7f9686d9d640 1 --2- 172.21.15.92:0/4103326831 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96881429f0 0x7f968813f0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9680009e50 tx=0x7f968002be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=757fa9a7cab9c355 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:07.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 1 -- 172.21.15.92:0/4103326831 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968002c9e0 con 0x7f96881429f0 2023-12-01T17:03:07.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: dump: 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: _finish_auth 0 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: _check_auth_tickets 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:07.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 1 -- 172.21.15.92:0/4103326831 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968002cb40 con 0x7f96881429f0 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 1 -- 172.21.15.92:0/4103326831 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96800343f0 con 0x7f96881429f0 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968559a640 10 monclient: dump: 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.197 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968cb0f640 5 monclient: authenticate success, global_id 4118 2023-12-01T17:03:07.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968cb0f640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:07.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.194+0000 7f968cb0f640 4 set_mon_vals no callback set 2023-12-01T17:03:07.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 10 monclient: shutdown 2023-12-01T17:03:07.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:07.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 -- 172.21.15.92:0/4103326831 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96881429f0 msgr2=0x7f968813f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 --2- 172.21.15.92:0/4103326831 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96881429f0 0x7f968813f0b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9680009e50 tx=0x7f968002be40 comp rx=0 tx=0).stop 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 -- 172.21.15.92:0/4103326831 shutdown_connections 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 --2- 172.21.15.92:0/4103326831 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96881429f0 0x7f968813f0b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 --2- 172.21.15.92:0/4103326831 >> v2:172.21.15.67:3300/0 conn(0x7f96880b21a0 0x7f96881424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 --2- 172.21.15.92:0/4103326831 >> v2:172.21.15.92:3300/0 conn(0x7f96880a8930 0x7f96880a8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 -- 172.21.15.92:0/4103326831 >> 172.21.15.92:0/4103326831 conn(0x7f96880b82f0 msgr2=0x7f96880b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 -- 172.21.15.92:0/4103326831 shutdown_connections 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.195+0000 7f968cb0f640 1 -- 172.21.15.92:0/4103326831 wait complete. 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 10 monclient: build_initial_monmap 2023-12-01T17:03:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 10 monclient: monmap: 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:07.202 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 1 librados: starting msgr at 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 1 librados: starting objecter 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 1 Processor -- start 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 1 -- start start 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 1 librados: setting wanted keys 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 1 librados: calling monclient init 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.196+0000 7f968cb0f640 10 monclient: init 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96880a8930 0x7f9688135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): picked mon.noname-c con 0x7f96880a8930 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96880b21a0 0x7f968813d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): picked mon.noname-a con 0x7f96880b21a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:07.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96881429f0 0x7f9688143470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): picked mon.noname-b con 0x7f96881429f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968813e430 con 0x7f96880b21a0 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968813e650 con 0x7f96881429f0 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968813e930 con 0x7f96880a8930 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.197+0000 7f968cb0f640 10 monclient(hunting): authenticate will time out at 1330.508218s 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9686d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96881429f0 0x7f9688143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f968659c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96880a8930 0x7f9688135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96880b21a0 0x7f968813d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96880b21a0 0x7f968813d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.92:52856/0 (socket says 172.21.15.92:52856) 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 1 -- 172.21.15.92:0/1879477318 learned_addr learned my addr 172.21.15.92:0/1879477318 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:07.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 10 monclient(hunting): get_auth_request con 0x7f96880b21a0 auth_method 0 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9686d9d640 10 monclient(hunting): get_auth_request con 0x7f96881429f0 auth_method 0 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9686d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9686d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9686d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f968659c640 10 monclient(hunting): get_auth_request con 0x7f96880a8930 auth_method 0 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f968659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f968659c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f968659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9685d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9686d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.198+0000 7f9686d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.199+0000 7f9686d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.199+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.199+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.199+0000 7f968659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 10 monclient(hunting): handle_auth_done global_id 4120 payload 824 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 1 -- 172.21.15.92:0/1879477318 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96880a8930 msgr2=0x7f9688135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 1 --2- 172.21.15.92:0/1879477318 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96880a8930 0x7f9688135ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 1 -- 172.21.15.92:0/1879477318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96880b21a0 msgr2=0x7f968813d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 1 --2- 172.21.15.92:0/1879477318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96880b21a0 0x7f968813d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 1 monclient: found mon.noname-b 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.201+0000 7f9686d9d640 1 -- 172.21.15.92:0/1879477318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9680009b30 con 0x7f96881429f0 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.203+0000 7f9686d9d640 1 --2- 172.21.15.92:0/1879477318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96881429f0 0x7f9688143470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f96880ab010 tx=0x7f9680004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.203+0000 7f9676ffd640 10 client.?.objecter ms_handle_connect 0x7f96881429f0 2023-12-01T17:03:07.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.203+0000 7f9676ffd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 1 -- 172.21.15.92:0/1879477318 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968003e450 con 0x7f96881429f0 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: dump: 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: _finish_auth 0 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 1 -- 172.21.15.92:0/1879477318 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968003e5d0 con 0x7f96881429f0 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 1 -- 172.21.15.92:0/1879477318 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968003e980 con 0x7f96881429f0 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 5 monclient: authenticate success, global_id 4120 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 10 monclient: _renew_subs 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968813ce70 con 0x7f96881429f0 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9684d99640 4 set_mon_vals no callback set 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f9676ffd640 10 monclient: dump: 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.210 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 10 monclient: _renew_subs 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968813df40 con 0x7f96881429f0 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.207+0000 7f968cb0f640 1 librados: init done 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.208+0000 7f968cb0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.208+0000 7f968cb0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.208+0000 7f968cb0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.208+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968813d0b0 con 0x7f96881429f0 2023-12-01T17:03:07.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.211+0000 7f9676ffd640 1 -- 172.21.15.92:0/1879477318 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9680051430 con 0x7f96881429f0 2023-12-01T17:03:07.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.211+0000 7f9676ffd640 1 -- 172.21.15.92:0/1879477318 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f968003ede0 con 0x7f96881429f0 2023-12-01T17:03:07.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.211+0000 7f9676ffd640 10 client.4120.objecter ms_dispatch 0x7f9688000b90 osd_map(4..4 src has 1..4) v4 2023-12-01T17:03:07.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.211+0000 7f9676ffd640 3 client.4120.objecter handle_osd_map got epochs [4,4] > 0 2023-12-01T17:03:07.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.211+0000 7f9676ffd640 3 client.4120.objecter handle_osd_map decoding full epoch 4 2023-12-01T17:03:07.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.211+0000 7f9676ffd640 20 client.4120.objecter dump_active .. 0 homeless 2023-12-01T17:03:07.307 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.307+0000 7f9676ffd640 1 -- 172.21.15.92:0/1879477318 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f968813d0b0 con 0x7f96881429f0 2023-12-01T17:03:07.307 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.307+0000 7f9676ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.307 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.307+0000 7f9676ffd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:07.323 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:07.323+0000 7fabd1a2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-01T17:03:07.365 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:07.365+0000 7f4645e3d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-12-01T17:03:07.379 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.379+0000 7f968cb0f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f72d775b-8e17-4768-87e3-6cc7a2bdb6a3", "id": 2}] 2023-12-01T17:03:07.379 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.379+0000 7f968cb0f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f72d775b-8e17-4768-87e3-6cc7a2bdb6a3", "id": 2}] 2023-12-01T17:03:07.379 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.379+0000 7f968cb0f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.379 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.379+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f72d775b-8e17-4768-87e3-6cc7a2bdb6a3", "id": 2} v 0) v1 -- 0x7f968813ea70 con 0x7f96881429f0 2023-12-01T17:03:07.452 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.452+0000 7f9676ffd640 1 -- 172.21.15.92:0/1879477318 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f72d775b-8e17-4768-87e3-6cc7a2bdb6a3", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f968813ea70 con 0x7f96881429f0 2023-12-01T17:03:07.452 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.452+0000 7f9676ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f72d775b-8e17-4768-87e3-6cc7a2bdb6a3", "id": 2}] 2023-12-01T17:03:07.452 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.452+0000 7f9676ffd640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:07.453 INFO:teuthology.orchestra.run.smithi092.stdout:2 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 10 librados: watch_flush enter 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 10 librados: watch_flush exit 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 20 client.4120.objecter shutdown clearing up homeless session... 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 10 client.4120.objecter successfully canceled tick 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 10 monclient: shutdown 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96881429f0 msgr2=0x7f9688143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 --2- 172.21.15.92:0/1879477318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96881429f0 0x7f9688143470 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f96880ab010 tx=0x7f9680004a40 comp rx=0 tx=0).stop 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 shutdown_connections 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 --2- 172.21.15.92:0/1879477318 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f96881429f0 0x7f9688143470 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.454 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 --2- 172.21.15.92:0/1879477318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96880b21a0 0x7f968813d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.455 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 --2- 172.21.15.92:0/1879477318 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f96880a8930 0x7f9688135ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.455 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 >> 172.21.15.92:0/1879477318 conn(0x7f96880b82f0 msgr2=0x7f9688073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:07.455 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 shutdown_connections 2023-12-01T17:03:07.455 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 -- 172.21.15.92:0/1879477318 wait complete. 2023-12-01T17:03:07.455 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.453+0000 7f968cb0f640 1 librados: shutdown 2023-12-01T17:03:07.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new ad66de04-1a5c-47a6-9e6e-68b141fac874 3 2023-12-01T17:03:07.466 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:07.467+0000 7fabd1a2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-01T17:03:07.508 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:07.508+0000 7f4645e3d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.563+0000 7f241b49b640 10 monclient: get_monmap_and_config 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.563+0000 7f241b49b640 10 monclient: build_initial_monmap 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.563+0000 7f241b49b640 10 monclient: monmap: 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:07.563417+0000 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:07.563417+0000 2023-12-01T17:03:07.563 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.565 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.563+0000 7f241b49b640 1 Processor -- start 2023-12-01T17:03:07.568 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.563+0000 7f241b49b640 1 -- start start 2023-12-01T17:03:07.568 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.563+0000 7f241b49b640 10 monclient: init 2023-12-01T17:03:07.568 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:07.568 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2414061900 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f24140713c0 0x7f24141430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient(hunting): picked mon.noname-c con 0x7f24140713c0 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2414072160 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24140a8cd0 con 0x7f2414061900 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2414070ea0 con 0x7f2414072160 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24140a8ad0 con 0x7f24140713c0 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f241b49b640 10 monclient(hunting): authenticate will time out at 1330.875175s 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.564+0000 7f2418a36640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f24140713c0 0x7f24141430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.569 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.565+0000 7f2419237640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2414061900 0x7f241413d000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.92:57036/0 (socket says 172.21.15.92:57036) 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.565+0000 7f2419237640 1 -- 172.21.15.92:0/3199638051 learned_addr learned my addr 172.21.15.92:0/3199638051 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.565+0000 7f2418a36640 10 monclient(hunting): get_auth_request con 0x7f24140713c0 auth_method 0 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.565+0000 7f2418a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.565+0000 7f2418a36640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.565+0000 7f2418a36640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.566+0000 7f2418a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.566+0000 7f2418a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.566+0000 7f2418a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f240bfff640 1 -- 172.21.15.92:0/3199638051 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158351628 0 0) 0x7f24140a8cd0 con 0x7f2414061900 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f240bfff640 10 monclient(hunting): _negotiate 2023-12-01T17:03:07.570 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f240bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f240bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f240bfff640 10 monclient(hunting): my global_id is 4128 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f240bfff640 1 -- 172.21.15.92:0/3199638051 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2400003520 con 0x7f2414061900 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f2418a36640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f2418a36640 1 -- 172.21.15.92:0/3199638051 >> v1:172.21.15.92:6789/0 conn(0x7f2414072160 legacy=0x7f241414a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f2418a36640 1 -- 172.21.15.92:0/3199638051 >> v1:172.21.15.67:6789/0 conn(0x7f2414061900 legacy=0x7f241413d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f2418a36640 10 monclient: _finish_hunting 0 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f2418a36640 1 monclient: found mon.noname-c 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f2418a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.567+0000 7f2418a36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.571 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.568+0000 7f2418a36640 1 -- 172.21.15.92:0/3199638051 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241413d710 con 0x7f24140713c0 2023-12-01T17:03:07.572 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.572+0000 7f2418a36640 1 --2- 172.21.15.92:0/3199638051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140713c0 0x7f24141430d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24100097a0 tx=0x7f241002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=edd904dd4e3c86a3 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:07.575 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 1 -- 172.21.15.92:0/3199638051 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241002e070 con 0x7f24140713c0 2023-12-01T17:03:07.575 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.575 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:07.575 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: dump: 2023-12-01T17:03:07.575 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.575 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: _finish_auth 0 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: _check_auth_tickets 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 1 -- 172.21.15.92:0/3199638051 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24100315a0 con 0x7f24140713c0 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 1 -- 172.21.15.92:0/3199638051 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2410031930 con 0x7f24140713c0 2023-12-01T17:03:07.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f241b49b640 5 monclient: authenticate success, global_id 4123 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f241b49b640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f241b49b640 4 set_mon_vals no callback set 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.575+0000 7f240bfff640 10 monclient: dump: 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.577 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.582 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 10 monclient: shutdown 2023-12-01T17:03:07.582 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 1 -- 172.21.15.92:0/3199638051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140713c0 msgr2=0x7f24141430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 1 --2- 172.21.15.92:0/3199638051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140713c0 0x7f24141430d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24100097a0 tx=0x7f241002b840 comp rx=0 tx=0).stop 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 1 -- 172.21.15.92:0/3199638051 shutdown_connections 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 1 --2- 172.21.15.92:0/3199638051 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f24140713c0 0x7f24141430d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 1 -- 172.21.15.92:0/3199638051 >> 172.21.15.92:0/3199638051 conn(0x7f24140b28c0 msgr2=0x7f2414060f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 1 -- 172.21.15.92:0/3199638051 shutdown_connections 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.576+0000 7f241b49b640 1 -- 172.21.15.92:0/3199638051 wait complete. 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.577+0000 7f241b49b640 10 monclient: build_initial_monmap 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.577+0000 7f241b49b640 10 monclient: monmap: 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:07.583 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.577+0000 7f241b49b640 1 librados: starting msgr at 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.577+0000 7f241b49b640 1 librados: starting objecter 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.577+0000 7f241b49b640 1 Processor -- start 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 -- start start 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 librados: setting wanted keys 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 librados: calling monclient init 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient: init 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:03:07.584 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2414061900 0x7f2414142a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.585 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): picked mon.noname-b con 0x7f2414061900 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:07.585 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f24140713c0 0x7f2414140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.585 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): picked mon.noname-c con 0x7f24140713c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:07.585 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2414072160 0x7f2414140670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.585 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): picked mon.noname-a con 0x7f2414072160 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24141d5210 con 0x7f2414072160 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241413fc40 con 0x7f2414061900 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2414140bb0 con 0x7f24140713c0 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f241b49b640 10 monclient(hunting): authenticate will time out at 1330.889293s 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.578+0000 7f2419237640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2414061900 0x7f2414142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f24140713c0 0x7f2414140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419a38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2414072160 0x7f2414140670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2414061900 0x7f2414142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43966/0 (socket says 172.21.15.92:43966) 2023-12-01T17:03:07.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 1 -- 172.21.15.92:0/4031449448 learned_addr learned my addr 172.21.15.92:0/4031449448 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 10 monclient(hunting): get_auth_request con 0x7f2414061900 auth_method 0 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 10 monclient(hunting): get_auth_request con 0x7f24140713c0 auth_method 0 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419a38640 10 monclient(hunting): get_auth_request con 0x7f2414072160 auth_method 0 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419a38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419a38640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419a38640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2419237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.579+0000 7f2418a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.580+0000 7f2419a38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.580+0000 7f2419a38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.580+0000 7f2419a38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 1 -- 172.21.15.92:0/4031449448 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f24140713c0 msgr2=0x7f2414140130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 1 --2- 172.21.15.92:0/4031449448 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f24140713c0 0x7f2414140130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 1 -- 172.21.15.92:0/4031449448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2414072160 msgr2=0x7f2414140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 1 --2- 172.21.15.92:0/4031449448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2414072160 0x7f2414140670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 10 monclient: _finish_hunting 0 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 1 monclient: found mon.noname-b 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:07.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.589 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.582+0000 7f2419237640 1 -- 172.21.15.92:0/4031449448 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2410009400 con 0x7f2414061900 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.584+0000 7f2419237640 1 --2- 172.21.15.92:0/4031449448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2414061900 0x7f2414142a50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f240400c500 tx=0x7f240400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.584+0000 7f2409ffb640 10 client.?.objecter ms_handle_connect 0x7f2414061900 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.584+0000 7f2409ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.587+0000 7f2409ffb640 1 -- 172.21.15.92:0/4031449448 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404005ce0 con 0x7f2414061900 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: dump: 2023-12-01T17:03:07.591 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: _finish_auth 0 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 1 -- 172.21.15.92:0/4031449448 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2404005e40 con 0x7f2414061900 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 5 monclient: authenticate success, global_id 4129 2023-12-01T17:03:07.592 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 10 monclient: _renew_subs 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24140af1e0 con 0x7f2414061900 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 1 -- 172.21.15.92:0/4031449448 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404007330 con 0x7f2414061900 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f2409ffb640 10 monclient: dump: 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f240b7fe640 4 set_mon_vals no callback set 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 10 monclient: _renew_subs 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241413ed50 con 0x7f2414061900 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.588+0000 7f241b49b640 1 librados: init done 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.589+0000 7f241b49b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.589+0000 7f241b49b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.589+0000 7f241b49b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.589+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24140708e0 con 0x7f2414061900 2023-12-01T17:03:07.594 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.591+0000 7f2409ffb640 1 -- 172.21.15.92:0/4031449448 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f240401c440 con 0x7f2414061900 2023-12-01T17:03:07.595 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.591+0000 7f2409ffb640 1 -- 172.21.15.92:0/4031449448 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f240401b840 con 0x7f2414061900 2023-12-01T17:03:07.595 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.591+0000 7f2409ffb640 10 client.4129.objecter ms_dispatch 0x7f2414000b90 osd_map(5..5 src has 1..5) v4 2023-12-01T17:03:07.595 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.591+0000 7f2409ffb640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2023-12-01T17:03:07.595 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.591+0000 7f2409ffb640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2023-12-01T17:03:07.595 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.591+0000 7f2409ffb640 20 client.4129.objecter dump_active .. 0 homeless 2023-12-01T17:03:07.595 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:07.593+0000 7fabd1a2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-01T17:03:07.641 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:07.641+0000 7fabd1a2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-01T17:03:07.686 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.686+0000 7f2409ffb640 1 -- 172.21.15.92:0/4031449448 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2404017070 con 0x7f2414061900 2023-12-01T17:03:07.686 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.686+0000 7f2409ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.686 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.686+0000 7f2409ffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:07.745 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:07.745+0000 7fabd1a2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-01T17:03:07.755 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.756+0000 7f241b49b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ad66de04-1a5c-47a6-9e6e-68b141fac874", "id": 3}] 2023-12-01T17:03:07.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.756+0000 7f241b49b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ad66de04-1a5c-47a6-9e6e-68b141fac874", "id": 3}] 2023-12-01T17:03:07.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.756+0000 7f241b49b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.756 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.756+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ad66de04-1a5c-47a6-9e6e-68b141fac874", "id": 3} v 0) v1 -- 0x7f241404e840 con 0x7f2414061900 2023-12-01T17:03:07.819 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.819+0000 7f2409ffb640 1 -- 172.21.15.92:0/4031449448 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad66de04-1a5c-47a6-9e6e-68b141fac874", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f240401ba50 con 0x7f2414061900 2023-12-01T17:03:07.819 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.819+0000 7f2409ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ad66de04-1a5c-47a6-9e6e-68b141fac874", "id": 3}] 2023-12-01T17:03:07.819 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.819+0000 7f2409ffb640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:07.819 INFO:teuthology.orchestra.run.smithi092.stdout:3 2023-12-01T17:03:07.820 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 10 librados: watch_flush enter 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 10 librados: watch_flush exit 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 20 client.4129.objecter shutdown clearing up homeless session... 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 10 client.4129.objecter successfully canceled tick 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 10 monclient: shutdown 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2414061900 msgr2=0x7f2414142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 --2- 172.21.15.92:0/4031449448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2414061900 0x7f2414142a50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f240400c500 tx=0x7f240400c9c0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 shutdown_connections 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 --2- 172.21.15.92:0/4031449448 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2414072160 0x7f2414140670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 --2- 172.21.15.92:0/4031449448 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f24140713c0 0x7f2414140130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 --2- 172.21.15.92:0/4031449448 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2414061900 0x7f2414142a50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 >> 172.21.15.92:0/4031449448 conn(0x7f24140b28c0 msgr2=0x7f2414136db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:07.821 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 shutdown_connections 2023-12-01T17:03:07.822 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 -- 172.21.15.92:0/4031449448 wait complete. 2023-12-01T17:03:07.822 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.820+0000 7f241b49b640 1 librados: shutdown 2023-12-01T17:03:07.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new dfce6366-008d-476d-b47a-f2d56833077c 4 2023-12-01T17:03:07.848 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:07.848+0000 7fabd1a2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-01T17:03:07.914 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:07.914+0000 7f4645e3d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-01T17:03:07.927 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.927+0000 7f600b33a640 10 monclient: get_monmap_and_config 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.927+0000 7f600b33a640 10 monclient: build_initial_monmap 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.927+0000 7f600b33a640 10 monclient: monmap: 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:07.928249+0000 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:07.928249+0000 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:07.928 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:07.929 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.929 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.928+0000 7f600b33a640 1 Processor -- start 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.928+0000 7f600b33a640 1 -- start start 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.928+0000 7f600b33a640 10 monclient: init 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60040a8930 0x7f60040a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient(hunting): picked mon.noname-c con 0x7f60040a8930 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60040b21a0 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f600413f0b0 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60040aa980 con 0x7f60040b21a0 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60040aabf0 con 0x7f600413f0b0 2023-12-01T17:03:07.930 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60040aaed0 con 0x7f60040a8930 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f600b33a640 10 monclient(hunting): authenticate will time out at 1331.240003s 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.929+0000 7f60090d6640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f60040a8930 0x7f60040a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.930+0000 7f60088d5640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f60040b21a0 0x7f60041424b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.92:57050/0 (socket says 172.21.15.92:57050) 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.930+0000 7f60088d5640 1 -- 172.21.15.92:0/3860327768 learned_addr learned my addr 172.21.15.92:0/3860327768 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.930+0000 7f60090d6640 10 monclient(hunting): get_auth_request con 0x7f60040a8930 auth_method 0 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.930+0000 7f60090d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.930+0000 7f60090d6640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.931 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.930+0000 7f60090d6640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.932 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.931+0000 7f60090d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.931+0000 7f60090d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.931+0000 7f60090d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f5ffbfff640 1 -- 172.21.15.92:0/3860327768 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618743519 0 0) 0x7f60040aa980 con 0x7f60040b21a0 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f5ffbfff640 10 monclient(hunting): _negotiate 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f5ffbfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f5ffbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f5ffbfff640 10 monclient(hunting): my global_id is 4134 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f5ffbfff640 1 -- 172.21.15.92:0/3860327768 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff0003520 con 0x7f60040b21a0 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 10 monclient(hunting): handle_auth_done global_id 4132 payload 274 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 1 -- 172.21.15.92:0/3860327768 >> v1:172.21.15.92:6789/0 conn(0x7f600413f0b0 legacy=0x7f600413f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 1 -- 172.21.15.92:0/3860327768 >> v1:172.21.15.67:6789/0 conn(0x7f60040b21a0 legacy=0x7f60041424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 10 monclient: _finish_hunting 0 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 1 monclient: found mon.noname-c 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:07.933 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.934 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.932+0000 7f60090d6640 1 -- 172.21.15.92:0/3860327768 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6004141f90 con 0x7f60040a8930 2023-12-01T17:03:07.935 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.935+0000 7f60090d6640 1 --2- 172.21.15.92:0/3860327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f60040a8d10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40097d0 tx=0x7f5ff402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=426387161ee6990e server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 1 -- 172.21.15.92:0/3860327768 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4031830 con 0x7f60040a8930 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: dump: 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.941 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: _finish_auth 0 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: _check_auth_tickets 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 1 -- 172.21.15.92:0/3860327768 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff4031990 con 0x7f60040a8930 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 1 -- 172.21.15.92:0/3860327768 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff4031d20 con 0x7f60040a8930 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f600b33a640 5 monclient: authenticate success, global_id 4132 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f600b33a640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f600b33a640 4 set_mon_vals no callback set 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.941+0000 7f5ffbfff640 10 monclient: dump: 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.943 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 10 monclient: shutdown 2023-12-01T17:03:07.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:07.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 1 -- 172.21.15.92:0/3860327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 msgr2=0x7f60040a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 1 --2- 172.21.15.92:0/3860327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f60040a8d10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5ff40097d0 tx=0x7f5ff402b8d0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 1 -- 172.21.15.92:0/3860327768 shutdown_connections 2023-12-01T17:03:07.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 1 --2- 172.21.15.92:0/3860327768 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f60040a8d10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 1 -- 172.21.15.92:0/3860327768 >> 172.21.15.92:0/3860327768 conn(0x7f60040b82f0 msgr2=0x7f60040b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 1 -- 172.21.15.92:0/3860327768 shutdown_connections 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.942+0000 7f600b33a640 1 -- 172.21.15.92:0/3860327768 wait complete. 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 10 monclient: build_initial_monmap 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 10 monclient: monmap: 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:07.948 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 1 librados: starting msgr at 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 1 librados: starting objecter 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 1 Processor -- start 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 1 -- start start 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 1 librados: setting wanted keys 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 1 librados: calling monclient init 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.943+0000 7f600b33a640 10 monclient: init 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f6004136c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.949 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): picked mon.noname-b con 0x7f60040a8930 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:07.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60040b21a0 0x7f6004139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): picked mon.noname-a con 0x7f60040b21a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:07.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f600413f0b0 0x7f600413a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:07.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): picked mon.noname-c con 0x7f600413f0b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:07.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6004145a10 con 0x7f60040b21a0 2023-12-01T17:03:07.950 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f600413a930 con 0x7f60040a8930 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6004137570 con 0x7f600413f0b0 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f600b33a640 10 monclient(hunting): authenticate will time out at 1331.254975s 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f60090d6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f6004136c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f60088d5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60040b21a0 0x7f6004139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f60098d7640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f600413f0b0 0x7f600413a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f60090d6640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f6004136c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43976/0 (socket says 172.21.15.92:43976) 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f60090d6640 1 -- 172.21.15.92:0/825581939 learned_addr learned my addr 172.21.15.92:0/825581939 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:07.951 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.944+0000 7f60090d6640 10 monclient(hunting): get_auth_request con 0x7f60040a8930 auth_method 0 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60090d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60090d6640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60090d6640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60088d5640 10 monclient(hunting): get_auth_request con 0x7f60040b21a0 auth_method 0 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60088d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60088d5640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60088d5640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60098d7640 10 monclient(hunting): get_auth_request con 0x7f600413f0b0 auth_method 0 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60098d7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60098d7640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60098d7640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60090d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60090d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60090d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60088d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60088d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.945+0000 7f60088d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.946+0000 7f60098d7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:07.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.946+0000 7f60098d7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:07.953 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.946+0000 7f60098d7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:07.955 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 1 -- 172.21.15.92:0/825581939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f600413f0b0 msgr2=0x7f600413a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 1 --2- 172.21.15.92:0/825581939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f600413f0b0 0x7f600413a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 1 -- 172.21.15.92:0/825581939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60040b21a0 msgr2=0x7f6004139eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 1 --2- 172.21.15.92:0/825581939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60040b21a0 0x7f6004139eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 10 monclient: _finish_hunting 0 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 1 monclient: found mon.noname-b 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:07.956 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.948+0000 7f60090d6640 1 -- 172.21.15.92:0/825581939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff4009400 con 0x7f60040a8930 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.950+0000 7f60090d6640 1 --2- 172.21.15.92:0/825581939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f6004136c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ff402bee0 tx=0x7f5ff40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.950+0000 7f5ff9ffb640 10 client.?.objecter ms_handle_connect 0x7f60040a8930 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.950+0000 7f5ff9ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 1 -- 172.21.15.92:0/825581939 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff403b9f0 con 0x7f60040a8930 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: dump: 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.957 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: _finish_auth 0 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 1 -- 172.21.15.92:0/825581939 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ff403bb70 con 0x7f60040a8930 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f600b33a640 5 monclient: authenticate success, global_id 4138 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f600b33a640 10 monclient: _renew_subs 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f600b33a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6004137e20 con 0x7f60040a8930 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 1 -- 172.21.15.92:0/825581939 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff40453f0 con 0x7f60040a8930 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:07.958 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ff9ffb640 10 monclient: dump: 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.953+0000 7f5ffb7fe640 4 set_mon_vals no callback set 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.954+0000 7f600b33a640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.954+0000 7f600b33a640 10 monclient: _renew_subs 2023-12-01T17:03:07.959 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.954+0000 7f600b33a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.960 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.954+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60041383e0 con 0x7f60040a8930 2023-12-01T17:03:07.960 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.954+0000 7f600b33a640 1 librados: init done 2023-12-01T17:03:07.960 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.955+0000 7f600b33a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.960 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.955+0000 7f600b33a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:07.960 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.955+0000 7f600b33a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:07.960 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:07.955+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f600404e840 con 0x7f60040a8930 2023-12-01T17:03:07.963 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:07.963+0000 7f4645e3d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-01T17:03:07.998 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:07.998+0000 7fabd1a2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-01T17:03:08.042 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:08.042+0000 7fabd1a2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-01T17:03:08.056 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 1 -- 172.21.15.92:0/825581939 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5ff4045850 con 0x7f60040a8930 2023-12-01T17:03:08.056 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 1 -- 172.21.15.92:0/825581939 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f5ff404ed10 con 0x7f60040a8930 2023-12-01T17:03:08.056 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 10 client.4138.objecter ms_dispatch 0x7f6004000b90 osd_map(6..6 src has 1..6) v4 2023-12-01T17:03:08.056 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2023-12-01T17:03:08.056 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2023-12-01T17:03:08.057 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 20 client.4138.objecter dump_active .. 0 homeless 2023-12-01T17:03:08.057 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 1 -- 172.21.15.92:0/825581939 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f5ff4035020 con 0x7f60040a8930 2023-12-01T17:03:08.057 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:08.057 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.056+0000 7f5ff9ffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:08.064 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:08.064+0000 7f4645e3d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-01T17:03:08.129 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.129+0000 7f600b33a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dfce6366-008d-476d-b47a-f2d56833077c", "id": 4}] 2023-12-01T17:03:08.129 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.129+0000 7f600b33a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dfce6366-008d-476d-b47a-f2d56833077c", "id": 4}] 2023-12-01T17:03:08.129 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.129+0000 7f600b33a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:08.129 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.129+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "dfce6366-008d-476d-b47a-f2d56833077c", "id": 4} v 0) v1 -- 0x7f60040b5d00 con 0x7f60040a8930 2023-12-01T17:03:08.163 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:08.163+0000 7f4645e3d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-01T17:03:08.193 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:08.193+0000 7fabd1a2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-01T17:03:08.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.194+0000 7f5ff9ffb640 1 -- 172.21.15.92:0/825581939 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dfce6366-008d-476d-b47a-f2d56833077c", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5ff404e7c0 con 0x7f60040a8930 2023-12-01T17:03:08.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.194+0000 7f5ff9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dfce6366-008d-476d-b47a-f2d56833077c", "id": 4}] 2023-12-01T17:03:08.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.194+0000 7f5ff9ffb640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:08.194 INFO:teuthology.orchestra.run.smithi092.stdout:4 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 10 librados: watch_flush enter 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 10 librados: watch_flush exit 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 20 client.4138.objecter shutdown clearing up homeless session... 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 10 client.4138.objecter successfully canceled tick 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 10 monclient: shutdown 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 msgr2=0x7f6004136c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 --2- 172.21.15.92:0/825581939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f6004136c00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5ff402bee0 tx=0x7f5ff40046c0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.196 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 shutdown_connections 2023-12-01T17:03:08.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 --2- 172.21.15.92:0/825581939 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f600413f0b0 0x7f600413a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 --2- 172.21.15.92:0/825581939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60040b21a0 0x7f6004139eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 --2- 172.21.15.92:0/825581939 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f60040a8930 0x7f6004136c00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 >> 172.21.15.92:0/825581939 conn(0x7f60040b82f0 msgr2=0x7f60040a9510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:08.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 shutdown_connections 2023-12-01T17:03:08.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 -- 172.21.15.92:0/825581939 wait complete. 2023-12-01T17:03:08.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.195+0000 7f600b33a640 1 librados: shutdown 2023-12-01T17:03:08.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 4fb3a3b6-1879-4cc9-9a36-07d8468de812 5 2023-12-01T17:03:08.307 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.307+0000 7fdc82f2c640 10 monclient: get_monmap_and_config 2023-12-01T17:03:08.307 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.307+0000 7fdc82f2c640 10 monclient: build_initial_monmap 2023-12-01T17:03:08.307 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.307+0000 7fdc82f2c640 10 monclient: monmap: 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:08.307867+0000 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:08.307867+0000 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:08.308 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:08.309 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:08.309 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:08.309 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.312 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 1 Processor -- start 2023-12-01T17:03:08.312 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 1 -- start start 2023-12-01T17:03:08.312 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient: init 2023-12-01T17:03:08.312 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc7c0a8930 0x7fdc7c0a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c0a8930 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdc7c0b21a0 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc7c13f0b0 0x7fdc7c13f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c13f0b0 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc7c0aa980 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c13fa20 con 0x7fdc7c0a8930 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7c0aabf0 con 0x7fdc7c13f0b0 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.308+0000 7fdc82f2c640 10 monclient(hunting): authenticate will time out at 1331.619682s 2023-12-01T17:03:08.313 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc814c9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc7c13f0b0 0x7fdc7c13f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc80cc8640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fdc7c0a8930 0x7fdc7c0a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc814c9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fdc7c13f0b0 0x7fdc7c13f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43980/0 (socket says 172.21.15.92:43980) 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc814c9640 1 -- 172.21.15.92:0/2525289720 learned_addr learned my addr 172.21.15.92:0/2525289720 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc814c9640 10 monclient(hunting): get_auth_request con 0x7fdc7c13f0b0 auth_method 0 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc814c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc814c9640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc814c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc80cc8640 10 monclient(hunting): get_auth_request con 0x7fdc7c0a8930 auth_method 0 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc80cc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc80cc8640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.309+0000 7fdc80cc8640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.314 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.310+0000 7fdc80cc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.310+0000 7fdc80cc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.310+0000 7fdc80cc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.310+0000 7fdc814c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.310+0000 7fdc814c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.310+0000 7fdc814c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 10 monclient(hunting): handle_auth_done global_id 4140 payload 274 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 1 -- 172.21.15.92:0/2525289720 >> v2:172.21.15.92:3300/0 conn(0x7fdc7c13f0b0 msgr2=0x7fdc7c13f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 1 --2- 172.21.15.92:0/2525289720 >> v2:172.21.15.92:3300/0 conn(0x7fdc7c13f0b0 0x7fdc7c13f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 1 -- 172.21.15.92:0/2525289720 >> v1:172.21.15.92:6789/0 conn(0x7fdc7c0b21a0 legacy=0x7fdc7c1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 10 monclient: _finish_hunting 0 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 1 monclient: found mon.noname-a 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:03:08.315 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.312+0000 7fdc80cc8640 1 -- 172.21.15.92:0/2525289720 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7c0aaed0 con 0x7fdc7c0a8930 2023-12-01T17:03:08.316 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.314+0000 7fdc80cc8640 1 --2- 172.21.15.92:0/2525289720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c0a8d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdc700097a0 tx=0x7fdc7002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440b928a17564649 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:08.316 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:08.314+0000 7f4645e3d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-01T17:03:08.317 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 1 -- 172.21.15.92:0/2525289720 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc7002e070 con 0x7fdc7c0a8930 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: dump: 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.318 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: _finish_auth 0 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 1 -- 172.21.15.92:0/2525289720 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc700315a0 con 0x7fdc7c0a8930 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 5 monclient: authenticate success, global_id 4140 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 1 -- 172.21.15.92:0/2525289720 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc70031930 con 0x7fdc7c0a8930 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 4 set_mon_vals no callback set 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc7b7fe640 10 monclient: dump: 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.319 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.320 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.320 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.320 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.320 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.320 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.320 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.320 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 10 monclient: shutdown 2023-12-01T17:03:08.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:08.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 -- 172.21.15.92:0/2525289720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 msgr2=0x7fdc7c0a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 --2- 172.21.15.92:0/2525289720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c0a8d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fdc700097a0 tx=0x7fdc7002b840 comp rx=0 tx=0).stop 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 -- 172.21.15.92:0/2525289720 shutdown_connections 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 --2- 172.21.15.92:0/2525289720 >> v2:172.21.15.92:3300/0 conn(0x7fdc7c13f0b0 0x7fdc7c13f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 --2- 172.21.15.92:0/2525289720 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c0a8d10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 -- 172.21.15.92:0/2525289720 >> 172.21.15.92:0/2525289720 conn(0x7fdc7c0b82f0 msgr2=0x7fdc7c0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 -- 172.21.15.92:0/2525289720 shutdown_connections 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.318+0000 7fdc82f2c640 1 -- 172.21.15.92:0/2525289720 wait complete. 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 10 monclient: build_initial_monmap 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 10 monclient: monmap: 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:08.324 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 1 librados: starting msgr at 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 1 librados: starting objecter 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 1 Processor -- start 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 1 -- start start 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 1 librados: setting wanted keys 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 1 librados: calling monclient init 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.319+0000 7fdc82f2c640 10 monclient: init 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c1372c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): picked mon.noname-a con 0x7fdc7c0a8930 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc7c0b21a0 0x7fdc7c1dafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): picked mon.noname-b con 0x7fdc7c0b21a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdc7c13f0b0 0x7fdc7c1db520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): picked mon.noname-c con 0x7fdc7c13f0b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c1dab50 con 0x7fdc7c0a8930 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c1dba60 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7c136290 con 0x7fdc7c13f0b0 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:08.326 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.320+0000 7fdc82f2c640 10 monclient(hunting): authenticate will time out at 1331.631379s 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc7c0b21a0 0x7fdc7c1dafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c1372c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc814c9640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdc7c13f0b0 0x7fdc7c1db520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc7c0b21a0 0x7fdc7c1dafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43982/0 (socket says 172.21.15.92:43982) 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 1 -- 172.21.15.92:0/487839549 learned_addr learned my addr 172.21.15.92:0/487839549 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 10 monclient(hunting): get_auth_request con 0x7fdc7c0b21a0 auth_method 0 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c1372c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.92:52906/0 (socket says 172.21.15.92:52906) 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 10 monclient(hunting): get_auth_request con 0x7fdc7c0a8930 auth_method 0 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc814c9640 10 monclient(hunting): get_auth_request con 0x7fdc7c13f0b0 auth_method 0 2023-12-01T17:03:08.327 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc814c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc814c9640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc814c9640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.321+0000 7fdc80cc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.322+0000 7fdc814c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.322+0000 7fdc814c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.328 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.322+0000 7fdc814c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.330 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 10 monclient(hunting): handle_auth_done global_id 4147 payload 824 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 1 -- 172.21.15.92:0/487839549 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdc7c13f0b0 msgr2=0x7fdc7c1db520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 1 --2- 172.21.15.92:0/487839549 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdc7c13f0b0 0x7fdc7c1db520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 1 -- 172.21.15.92:0/487839549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 msgr2=0x7fdc7c1372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 1 --2- 172.21.15.92:0/487839549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c1372c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 1 monclient: found mon.noname-b 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.324+0000 7fdc7bfff640 1 -- 172.21.15.92:0/487839549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc70009400 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.331 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.326+0000 7fdc7bfff640 1 --2- 172.21.15.92:0/487839549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc7c0b21a0 0x7fdc7c1dafe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800c400 tx=0x7fdc6800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.326+0000 7fdc797fa640 10 client.?.objecter ms_handle_connect 0x7fdc7c0b21a0 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.326+0000 7fdc797fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 1 -- 172.21.15.92:0/487839549 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc680052d0 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: dump: 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.332 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.333 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.333 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.333 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.333 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.334 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: _finish_auth 0 2023-12-01T17:03:08.334 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: _check_auth_tickets 2023-12-01T17:03:08.334 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:08.334 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 1 -- 172.21.15.92:0/487839549 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc68005450 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 5 monclient: authenticate success, global_id 4147 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 10 monclient: _renew_subs 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7c136570 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 1 -- 172.21.15.92:0/487839549 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc68007260 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc797fa640 10 monclient: dump: 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.335 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc7affd640 4 set_mon_vals no callback set 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 10 monclient: _renew_subs 2023-12-01T17:03:08.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:08.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc7c136b30 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.330+0000 7fdc82f2c640 1 librados: init done 2023-12-01T17:03:08.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.331+0000 7fdc82f2c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:08.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.331+0000 7fdc82f2c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:08.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.331+0000 7fdc82f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:08.337 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.331+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc7c1dad00 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.358 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:08.359+0000 7f4645e3d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-01T17:03:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 1 -- 172.21.15.92:0/487839549 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fdc680067d0 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 1 -- 172.21.15.92:0/487839549 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fdc68011590 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 10 client.4147.objecter ms_dispatch 0x7fdc7c000b90 osd_map(7..7 src has 1..7) v4 2023-12-01T17:03:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2023-12-01T17:03:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2023-12-01T17:03:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 20 client.4147.objecter dump_active .. 0 homeless 2023-12-01T17:03:08.428 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 1 -- 172.21.15.92:0/487839549 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fdc68020060 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.429 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:08.429 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.428+0000 7fdc797fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:08.500 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.500+0000 7fdc82f2c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4fb3a3b6-1879-4cc9-9a36-07d8468de812", "id": 5}] 2023-12-01T17:03:08.500 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.500+0000 7fdc82f2c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4fb3a3b6-1879-4cc9-9a36-07d8468de812", "id": 5}] 2023-12-01T17:03:08.500 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.500+0000 7fdc82f2c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:08.500 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.500+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4fb3a3b6-1879-4cc9-9a36-07d8468de812", "id": 5} v 0) v1 -- 0x7fdc7c04e840 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.501 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:08.501+0000 7f4645e3d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-01T17:03:08.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.585+0000 7fdc797fa640 1 -- 172.21.15.92:0/487839549 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4fb3a3b6-1879-4cc9-9a36-07d8468de812", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdc68006c10 con 0x7fdc7c0b21a0 2023-12-01T17:03:08.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.585+0000 7fdc797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4fb3a3b6-1879-4cc9-9a36-07d8468de812", "id": 5}] 2023-12-01T17:03:08.586 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.585+0000 7fdc797fa640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:08.586 INFO:teuthology.orchestra.run.smithi092.stdout:5 2023-12-01T17:03:08.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 10 librados: watch_flush enter 2023-12-01T17:03:08.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 10 librados: watch_flush exit 2023-12-01T17:03:08.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 20 client.4147.objecter shutdown clearing up homeless session... 2023-12-01T17:03:08.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 10 client.4147.objecter successfully canceled tick 2023-12-01T17:03:08.587 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 10 monclient: shutdown 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc7c0b21a0 msgr2=0x7fdc7c1dafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 --2- 172.21.15.92:0/487839549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc7c0b21a0 0x7fdc7c1dafe0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdc6800c400 tx=0x7fdc6800c8c0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 shutdown_connections 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 --2- 172.21.15.92:0/487839549 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fdc7c13f0b0 0x7fdc7c1db520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 --2- 172.21.15.92:0/487839549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fdc7c0b21a0 0x7fdc7c1dafe0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 --2- 172.21.15.92:0/487839549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdc7c0a8930 0x7fdc7c1372c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 >> 172.21.15.92:0/487839549 conn(0x7fdc7c0b82f0 msgr2=0x7fdc7c0b2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 shutdown_connections 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 -- 172.21.15.92:0/487839549 wait complete. 2023-12-01T17:03:08.588 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.586+0000 7fdc82f2c640 1 librados: shutdown 2023-12-01T17:03:08.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 76c95f42-d993-4ac3-85b3-f6357c571f1c 6 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.699+0000 7fb913a07640 10 monclient: get_monmap_and_config 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.699+0000 7fb913a07640 10 monclient: build_initial_monmap 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.699+0000 7fb913a07640 10 monclient: monmap: 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:08.699687+0000 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:08.699687+0000 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:08.699 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:08.700 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:08.700 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:08.700 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:08.700 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.699+0000 7fb913a07640 1 Processor -- start 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 1 -- start start 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient: init 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb90c0a8930 0x7fb90c0a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient(hunting): picked mon.noname-c con 0x7fb90c0a8930 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:08.701 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb90c0b21a0 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb90c13f0b0 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c0aa980 con 0x7fb90c0b21a0 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb90c0aabf0 con 0x7fb90c13f0b0 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb90c0aaed0 con 0x7fb90c0a8930 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.700+0000 7fb913a07640 10 monclient(hunting): authenticate will time out at 1332.011378s 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.701+0000 7fb9117a3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb90c0a8930 0x7fb90c0a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.701+0000 7fb9117a3640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb90c0a8930 0x7fb90c0a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:43984/0 (socket says 172.21.15.92:43984) 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.701+0000 7fb9117a3640 1 -- 172.21.15.92:0/2848169392 learned_addr learned my addr 172.21.15.92:0/2848169392 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.701+0000 7fb9117a3640 10 monclient(hunting): get_auth_request con 0x7fb90c0a8930 auth_method 0 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.701+0000 7fb9117a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.701+0000 7fb9117a3640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.701+0000 7fb9117a3640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.703 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.703+0000 7fb9117a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.703 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.703+0000 7fb9117a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.703 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.703+0000 7fb9117a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.704 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.704+0000 7fb903fff640 1 -- 172.21.15.92:0/2848169392 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224762475 0 0) 0x7fb90c0aa980 con 0x7fb90c0b21a0 2023-12-01T17:03:08.704 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.704+0000 7fb903fff640 10 monclient(hunting): _negotiate 2023-12-01T17:03:08.704 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.704+0000 7fb903fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.704 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.704+0000 7fb903fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.704 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.704+0000 7fb903fff640 10 monclient(hunting): my global_id is 4146 2023-12-01T17:03:08.704 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.704+0000 7fb903fff640 1 -- 172.21.15.92:0/2848169392 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8f8003520 con 0x7fb90c0b21a0 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 1 -- 172.21.15.92:0/2848169392 >> v1:172.21.15.92:6789/0 conn(0x7fb90c13f0b0 legacy=0x7fb90c13f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 1 -- 172.21.15.92:0/2848169392 >> v1:172.21.15.67:6789/0 conn(0x7fb90c0b21a0 legacy=0x7fb90c1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 10 monclient: _finish_hunting 0 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 1 monclient: found mon.noname-c 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:03:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.705+0000 7fb9117a3640 1 -- 172.21.15.92:0/2848169392 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb90c141f90 con 0x7fb90c0a8930 2023-12-01T17:03:08.710 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.710+0000 7fb9117a3640 1 --2- 172.21.15.92:0/2848169392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0a8930 0x7fb90c0a8d10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb904009e80 tx=0x7fb90402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=294cced33e0a6fa7 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:08.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 1 -- 172.21.15.92:0/2848169392 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90402caf0 con 0x7fb90c0a8930 2023-12-01T17:03:08.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:08.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: dump: 2023-12-01T17:03:08.715 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.715 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: _finish_auth 0 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: _check_auth_tickets 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 1 -- 172.21.15.92:0/2848169392 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90402cc50 con 0x7fb90c0a8930 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 1 -- 172.21.15.92:0/2848169392 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9040343f0 con 0x7fb90c0a8930 2023-12-01T17:03:08.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb913a07640 5 monclient: authenticate success, global_id 4150 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb913a07640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb913a07640 4 set_mon_vals no callback set 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.715+0000 7fb903fff640 10 monclient: dump: 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.717 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.720 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 10 monclient: shutdown 2023-12-01T17:03:08.720 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:08.720 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 1 -- 172.21.15.92:0/2848169392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0a8930 msgr2=0x7fb90c0a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.720 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 1 --2- 172.21.15.92:0/2848169392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0a8930 0x7fb90c0a8d10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb904009e80 tx=0x7fb90402bf50 comp rx=0 tx=0).stop 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 1 -- 172.21.15.92:0/2848169392 shutdown_connections 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 1 --2- 172.21.15.92:0/2848169392 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0a8930 0x7fb90c0a8d10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 1 -- 172.21.15.92:0/2848169392 >> 172.21.15.92:0/2848169392 conn(0x7fb90c0b82f0 msgr2=0x7fb90c0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 1 -- 172.21.15.92:0/2848169392 shutdown_connections 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.716+0000 7fb913a07640 1 -- 172.21.15.92:0/2848169392 wait complete. 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 10 monclient: build_initial_monmap 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 10 monclient: monmap: 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:08.721 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 1 librados: starting msgr at 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 1 librados: starting objecter 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 1 Processor -- start 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 1 -- start start 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 1 librados: setting wanted keys 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 1 librados: calling monclient init 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.717+0000 7fb913a07640 10 monclient: init 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb90c0a8930 0x7fb90c145e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): picked mon.noname-c con 0x7fb90c0a8930 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0b21a0 0x7fb90c1454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.722 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): picked mon.noname-b con 0x7fb90c0b21a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb90c13f0b0 0x7fb90c1333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): picked mon.noname-a con 0x7fb90c13f0b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c132a80 con 0x7fb90c13f0b0 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c1459e0 con 0x7fb90c0b21a0 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb90c132cd0 con 0x7fb90c0a8930 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb913a07640 10 monclient(hunting): authenticate will time out at 1332.029211s 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.718+0000 7fb9117a3640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb90c0a8930 0x7fb90c145e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb90c13f0b0 0x7fb90c1333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb90c0a8930 0x7fb90c145e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.92:54556/0 (socket says 172.21.15.92:54556) 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 1 -- 172.21.15.92:0/3381341779 learned_addr learned my addr 172.21.15.92:0/3381341779 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 10 monclient(hunting): get_auth_request con 0x7fb90c0a8930 auth_method 0 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 10 monclient(hunting): get_auth_request con 0x7fb90c13f0b0 auth_method 0 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb910fa2640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0b21a0 0x7fb90c1454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb910fa2640 10 monclient(hunting): get_auth_request con 0x7fb90c0b21a0 auth_method 0 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb910fa2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb910fa2640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb910fa2640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.724 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb9117a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.719+0000 7fb911fa4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.720+0000 7fb910fa2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:08.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.720+0000 7fb910fa2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:08.725 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.720+0000 7fb910fa2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 10 monclient(hunting): handle_auth_done global_id 4133 payload 824 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 1 -- 172.21.15.92:0/3381341779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0b21a0 msgr2=0x7fb90c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0b21a0 0x7fb90c1454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 1 -- 172.21.15.92:0/3381341779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb90c13f0b0 msgr2=0x7fb90c1333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb90c13f0b0 0x7fb90c1333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 10 monclient: _finish_hunting 0 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 1 monclient: found mon.noname-c 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.722+0000 7fb9117a3640 1 -- 172.21.15.92:0/3381341779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904009b30 con 0x7fb90c0a8930 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.725+0000 7fb9117a3640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb90c0a8930 0x7fb90c145e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb904036040 tx=0x7fb904004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.725+0000 7fb901ffb640 10 client.?.objecter ms_handle_connect 0x7fb90c0a8930 2023-12-01T17:03:08.726 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.725+0000 7fb901ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:08.728 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 1 -- 172.21.15.92:0/3381341779 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904009510 con 0x7fb90c0a8930 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: dump: 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: _finish_auth 0 2023-12-01T17:03:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 5 monclient: authenticate success, global_id 4133 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 1 -- 172.21.15.92:0/3381341779 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90403e450 con 0x7fb90c0a8930 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 1 -- 172.21.15.92:0/3381341779 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90403e800 con 0x7fb90c0a8930 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 10 monclient: _renew_subs 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90c133910 con 0x7fb90c0a8930 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb901ffb640 10 monclient: dump: 2023-12-01T17:03:08.730 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:08.731 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:08.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb9037fe640 4 set_mon_vals no callback set 2023-12-01T17:03:08.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:08.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 10 monclient: _renew_subs 2023-12-01T17:03:08.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:08.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90c12c880 con 0x7fb90c0a8930 2023-12-01T17:03:08.732 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.728+0000 7fb913a07640 1 librados: init done 2023-12-01T17:03:08.733 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.729+0000 7fb913a07640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:08.733 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.729+0000 7fb913a07640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:08.733 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.729+0000 7fb913a07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:08.733 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.729+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb90c133c40 con 0x7fb90c0a8930 2023-12-01T17:03:08.905 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.905+0000 7fb901ffb640 1 -- 172.21.15.92:0/3381341779 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb90403ec60 con 0x7fb90c0a8930 2023-12-01T17:03:08.912 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 1 -- 172.21.15.92:0/3381341779 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fb904051a20 con 0x7fb90c0a8930 2023-12-01T17:03:08.912 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 10 client.4133.objecter ms_dispatch 0x7fb90c000b90 osd_map(8..8 src has 1..8) v4 2023-12-01T17:03:08.912 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 3 client.4133.objecter handle_osd_map got epochs [8,8] > 0 2023-12-01T17:03:08.912 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 3 client.4133.objecter handle_osd_map decoding full epoch 8 2023-12-01T17:03:08.912 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 20 client.4133.objecter dump_active .. 0 homeless 2023-12-01T17:03:08.913 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 1 -- 172.21.15.92:0/3381341779 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fb904043020 con 0x7fb90c0a8930 2023-12-01T17:03:08.913 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:08.913 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.912+0000 7fb901ffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:08.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.984+0000 7fb913a07640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "76c95f42-d993-4ac3-85b3-f6357c571f1c", "id": 6}] 2023-12-01T17:03:08.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.984+0000 7fb913a07640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "76c95f42-d993-4ac3-85b3-f6357c571f1c", "id": 6}] 2023-12-01T17:03:08.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.984+0000 7fb913a07640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:08.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:08.984+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd new", "uuid": "76c95f42-d993-4ac3-85b3-f6357c571f1c", "id": 6} v 0) v1 -- 0x7fb90c145b20 con 0x7fb90c0a8930 2023-12-01T17:03:09.081 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:09.081+0000 7fabd1a2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-01T17:03:09.090 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.090+0000 7fb901ffb640 1 -- 172.21.15.92:0/3381341779 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76c95f42-d993-4ac3-85b3-f6357c571f1c", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb90c145b20 con 0x7fb90c0a8930 2023-12-01T17:03:09.090 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.090+0000 7fb901ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "76c95f42-d993-4ac3-85b3-f6357c571f1c", "id": 6}] 2023-12-01T17:03:09.090 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.090+0000 7fb901ffb640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:09.091 INFO:teuthology.orchestra.run.smithi092.stdout:6 2023-12-01T17:03:09.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 10 librados: watch_flush enter 2023-12-01T17:03:09.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 10 librados: watch_flush exit 2023-12-01T17:03:09.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 20 client.4133.objecter shutdown clearing up homeless session... 2023-12-01T17:03:09.092 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 10 client.4133.objecter successfully canceled tick 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 10 monclient: shutdown 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb90c0a8930 msgr2=0x7fb90c145e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb90c0a8930 0x7fb90c145e70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb904036040 tx=0x7fb904004a40 comp rx=0 tx=0).stop 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 shutdown_connections 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb90c13f0b0 0x7fb90c1333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb90c0b21a0 0x7fb90c1454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 --2- 172.21.15.92:0/3381341779 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb90c0a8930 0x7fb90c145e70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 >> 172.21.15.92:0/3381341779 conn(0x7fb90c0b82f0 msgr2=0x7fb90c0b63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 shutdown_connections 2023-12-01T17:03:09.093 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 -- 172.21.15.92:0/3381341779 wait complete. 2023-12-01T17:03:09.094 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.091+0000 7fb913a07640 1 librados: shutdown 2023-12-01T17:03:09.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd new 465c0392-a311-4499-b75a-7b5ad156d99e 7 2023-12-01T17:03:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.201+0000 7eff1fb73640 10 monclient: get_monmap_and_config 2023-12-01T17:03:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.201+0000 7eff1fb73640 10 monclient: build_initial_monmap 2023-12-01T17:03:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.201+0000 7eff1fb73640 10 monclient: monmap: 2023-12-01T17:03:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:03:09.202073+0000 2023-12-01T17:03:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:03:09.202073+0000 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.202+0000 7eff1fb73640 1 Processor -- start 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.202+0000 7eff1fb73640 1 -- start start 2023-12-01T17:03:09.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.202+0000 7eff1fb73640 10 monclient: init 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180a8930 0x7eff18145920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient(hunting): picked mon.noname-b con 0x7eff180a8930 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff180b2210 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff180a9b90 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff180aac10 con 0x7eff180a9b90 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff18145490 con 0x7eff180b2210 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff180aae60 con 0x7eff180a8930 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1fb73640 10 monclient(hunting): authenticate will time out at 1332.513842s 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1d10e640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7eff180b2210 0x7eff1813f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:41286/0 (socket says 172.21.15.92:41286) 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1d90f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180a8930 0x7eff18145920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.203+0000 7eff1d10e640 1 -- 172.21.15.92:0/2068743613 learned_addr learned my addr 172.21.15.92:0/2068743613 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.204+0000 7eff1d90f640 10 monclient(hunting): get_auth_request con 0x7eff180a8930 auth_method 0 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.204+0000 7eff1d90f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.204+0000 7eff1d90f640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.204+0000 7eff1d90f640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.204+0000 7eff1d90f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.204+0000 7eff1d90f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:09.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.204+0000 7eff1d90f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1c90d640 1 -- 172.21.15.92:0/2068743613 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789488992 0 0) 0x7eff18145490 con 0x7eff180b2210 2023-12-01T17:03:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1c90d640 10 monclient(hunting): _negotiate 2023-12-01T17:03:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1c90d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1c90d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1c90d640 10 monclient(hunting): my global_id is 4159 2023-12-01T17:03:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1c90d640 1 -- 172.21.15.92:0/2068743613 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff04003520 con 0x7eff180b2210 2023-12-01T17:03:09.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 1 -- 172.21.15.92:0/2068743613 >> v1:172.21.15.92:6789/0 conn(0x7eff180b2210 legacy=0x7eff1813f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 1 -- 172.21.15.92:0/2068743613 >> v1:172.21.15.67:6789/0 conn(0x7eff180a9b90 legacy=0x7eff1814c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 10 monclient: _finish_hunting 0 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 1 monclient: found mon.noname-b 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.206+0000 7eff1d90f640 1 -- 172.21.15.92:0/2068743613 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff1813f720 con 0x7eff180a8930 2023-12-01T17:03:09.208 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.209+0000 7eff1d90f640 1 --2- 172.21.15.92:0/2068743613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180a8930 0x7eff18145920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff140097d0 tx=0x7eff1402b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fba5e402a1bb959 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:09.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 1 -- 172.21.15.92:0/2068743613 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14031770 con 0x7eff180a8930 2023-12-01T17:03:09.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: dump: 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:09.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: _finish_auth 0 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: _check_auth_tickets 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 5 monclient: authenticate success, global_id 4136 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 1 -- 172.21.15.92:0/2068743613 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff140318d0 con 0x7eff180a8930 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 4 set_mon_vals no callback set 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 1 -- 172.21.15.92:0/2068743613 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14031c60 con 0x7eff180a8930 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1c90d640 10 monclient: dump: 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 10 monclient: shutdown 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:09.216 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 1 -- 172.21.15.92:0/2068743613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180a8930 msgr2=0x7eff18145920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.217 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 1 --2- 172.21.15.92:0/2068743613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180a8930 0x7eff18145920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff140097d0 tx=0x7eff1402b810 comp rx=0 tx=0).stop 2023-12-01T17:03:09.217 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 1 -- 172.21.15.92:0/2068743613 shutdown_connections 2023-12-01T17:03:09.217 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 1 --2- 172.21.15.92:0/2068743613 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180a8930 0x7eff18145920 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.217 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.213+0000 7eff1fb73640 1 -- 172.21.15.92:0/2068743613 >> 172.21.15.92:0/2068743613 conn(0x7eff180b8380 msgr2=0x7eff180b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:09.217 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.214+0000 7eff1fb73640 1 -- 172.21.15.92:0/2068743613 shutdown_connections 2023-12-01T17:03:09.217 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.214+0000 7eff1fb73640 1 -- 172.21.15.92:0/2068743613 wait complete. 2023-12-01T17:03:09.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.214+0000 7eff1fb73640 10 monclient: build_initial_monmap 2023-12-01T17:03:09.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.214+0000 7eff1fb73640 10 monclient: monmap: 2023-12-01T17:03:09.220 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 0 2023-12-01T17:03:09.220 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 0.000000 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:created 0.000000 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.214+0000 7eff1fb73640 1 librados: starting msgr at 2023-12-01T17:03:09.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.214+0000 7eff1fb73640 1 librados: starting objecter 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 Processor -- start 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 -- start start 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 librados: setting wanted keys 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 librados: calling monclient init 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient: init 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff180a8930 0x7eff180b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): picked mon.noname-b con 0x7eff180a8930 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff180a9b90 0x7eff181daa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): picked mon.noname-a con 0x7eff180a9b90 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:09.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180b2210 0x7eff181358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): picked mon.noname-c con 0x7eff180b2210 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff181db430 con 0x7eff180a9b90 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff180b1180 con 0x7eff180a8930 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff181daf80 con 0x7eff180b2210 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.215+0000 7eff1fb73640 10 monclient(hunting): authenticate will time out at 1332.526324s 2023-12-01T17:03:09.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff180a8930 0x7eff180b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180b2210 0x7eff181358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180b2210 0x7eff181358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.92:54574/0 (socket says 172.21.15.92:54574) 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 1 -- 172.21.15.92:0/2089428934 learned_addr learned my addr 172.21.15.92:0/2089428934 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 10 monclient(hunting): get_auth_request con 0x7eff180b2210 auth_method 0 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 10 monclient(hunting): get_auth_request con 0x7eff180a8930 auth_method 0 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1e110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:09.224 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.216+0000 7eff1d90f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 10 monclient(hunting): handle_auth_done global_id 4162 payload 824 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 1 -- 172.21.15.92:0/2089428934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180b2210 msgr2=0x7eff181358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 1 --2- 172.21.15.92:0/2089428934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180b2210 0x7eff181358a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 1 -- 172.21.15.92:0/2089428934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff180a9b90 msgr2=0x7eff181daa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 1 --2- 172.21.15.92:0/2089428934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff180a9b90 0x7eff181daa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 10 monclient: _finish_hunting 0 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 1 monclient: found mon.noname-b 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:09.225 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.219+0000 7eff1d90f640 1 -- 172.21.15.92:0/2089428934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff14009400 con 0x7eff180a8930 2023-12-01T17:03:09.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.221+0000 7eff1d90f640 1 --2- 172.21.15.92:0/2089428934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff180a8930 0x7eff180b06e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff14033000 tx=0x7eff140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:09.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.221+0000 7eff0e7fc640 10 client.?.objecter ms_handle_connect 0x7eff180a8930 2023-12-01T17:03:09.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.222+0000 7eff0e7fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:09.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff0e7fc640 1 -- 172.21.15.92:0/2089428934 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14032810 con 0x7eff180a8930 2023-12-01T17:03:09.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.227 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff0e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff0e7fc640 10 monclient: dump: 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:09.228 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff0e7fc640 10 monclient: _finish_auth 0 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff0e7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 5 monclient: authenticate success, global_id 4162 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 10 monclient: _renew_subs 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:09.229 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff181db1a0 con 0x7eff180a8930 2023-12-01T17:03:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 10 monclient: _renew_subs 2023-12-01T17:03:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff18135fc0 con 0x7eff180a8930 2023-12-01T17:03:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.224+0000 7eff1fb73640 1 librados: init done 2023-12-01T17:03:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.225+0000 7eff1fb73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.225+0000 7eff1fb73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.225+0000 7eff1fb73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.225+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff180b4010 con 0x7eff180a8930 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.227+0000 7eff0e7fc640 1 -- 172.21.15.92:0/2089428934 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff14032970 con 0x7eff180a8930 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.227+0000 7eff0e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.227+0000 7eff0e7fc640 1 -- 172.21.15.92:0/2089428934 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff14032ce0 con 0x7eff180a8930 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.227+0000 7eff0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.227+0000 7eff0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.227+0000 7eff0f7fe640 4 set_mon_vals no callback set 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.227+0000 7eff0e7fc640 10 monclient: dump: 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:epoch 1 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.231 INFO:teuthology.orchestra.run.smithi092.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.232 INFO:teuthology.orchestra.run.smithi092.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.232 INFO:teuthology.orchestra.run.smithi092.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.232 INFO:teuthology.orchestra.run.smithi092.stderr:election_strategy: 1 2023-12-01T17:03:09.232 INFO:teuthology.orchestra.run.smithi092.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.232 INFO:teuthology.orchestra.run.smithi092.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.232 INFO:teuthology.orchestra.run.smithi092.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.232 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:03:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 1 -- 172.21.15.92:0/2089428934 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7eff1404e7a0 con 0x7eff180a8930 2023-12-01T17:03:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 1 -- 172.21.15.92:0/2089428934 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7eff1403ba50 con 0x7eff180a8930 2023-12-01T17:03:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 10 client.4162.objecter ms_dispatch 0x7eff18000b90 osd_map(9..9 src has 1..9) v4 2023-12-01T17:03:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2023-12-01T17:03:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2023-12-01T17:03:09.329 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 20 client.4162.objecter dump_active .. 0 homeless 2023-12-01T17:03:09.330 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 1 -- 172.21.15.92:0/2089428934 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7eff1403bc80 con 0x7eff180a8930 2023-12-01T17:03:09.330 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:09.330 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.329+0000 7eff0e7fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:09.348 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:09.347+0000 7f4645e3d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-01T17:03:09.392 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:09.392+0000 7fabd1a2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-01T17:03:09.401 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.402+0000 7eff1fb73640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "465c0392-a311-4499-b75a-7b5ad156d99e", "id": 7}] 2023-12-01T17:03:09.401 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.402+0000 7eff1fb73640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "465c0392-a311-4499-b75a-7b5ad156d99e", "id": 7}] 2023-12-01T17:03:09.402 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.402+0000 7eff1fb73640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:09.402 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.402+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd new", "uuid": "465c0392-a311-4499-b75a-7b5ad156d99e", "id": 7} v 0) v1 -- 0x7eff181db650 con 0x7eff180a8930 2023-12-01T17:03:09.453 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:09.453+0000 7fabd1a2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-01T17:03:09.502 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.502+0000 7eff0e7fc640 1 -- 172.21.15.92:0/2089428934 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "465c0392-a311-4499-b75a-7b5ad156d99e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7eff1403be20 con 0x7eff180a8930 2023-12-01T17:03:09.502 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.502+0000 7eff0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "465c0392-a311-4499-b75a-7b5ad156d99e", "id": 7}] 2023-12-01T17:03:09.502 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.502+0000 7eff0e7fc640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:09.502 INFO:teuthology.orchestra.run.smithi092.stdout:7 2023-12-01T17:03:09.503 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 10 librados: watch_flush enter 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 10 librados: watch_flush exit 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 20 client.4162.objecter shutdown clearing up homeless session... 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 10 client.4162.objecter successfully canceled tick 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 10 monclient: shutdown 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff180a8930 msgr2=0x7eff180b06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 --2- 172.21.15.92:0/2089428934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff180a8930 0x7eff180b06e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7eff14033000 tx=0x7eff140046c0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 shutdown_connections 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 --2- 172.21.15.92:0/2089428934 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7eff180b2210 0x7eff181358a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 --2- 172.21.15.92:0/2089428934 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7eff180a9b90 0x7eff181daa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 --2- 172.21.15.92:0/2089428934 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7eff180a8930 0x7eff180b06e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 >> 172.21.15.92:0/2089428934 conn(0x7eff180b8380 msgr2=0x7eff18139b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:09.504 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 shutdown_connections 2023-12-01T17:03:09.505 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 -- 172.21.15.92:0/2089428934 wait complete. 2023-12-01T17:03:09.505 INFO:teuthology.orchestra.run.smithi092.stderr:2023-12-01T17:03:09.503+0000 7eff1fb73640 1 librados: shutdown 2023-12-01T17:03:09.513 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.514 INFO:tasks.ceph.osd.0:Restarting daemon 2023-12-01T17:03:09.514 DEBUG:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-12-01T17:03:09.516 INFO:tasks.ceph.osd.0:Started 2023-12-01T17:03:09.516 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.516 INFO:tasks.ceph.osd.1:Restarting daemon 2023-12-01T17:03:09.517 DEBUG:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-12-01T17:03:09.520 INFO:tasks.ceph.osd.1:Started 2023-12-01T17:03:09.520 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.520 INFO:tasks.ceph.osd.2:Restarting daemon 2023-12-01T17:03:09.520 DEBUG:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-12-01T17:03:09.524 INFO:tasks.ceph.osd.2:Started 2023-12-01T17:03:09.524 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.524 INFO:tasks.ceph.osd.3:Restarting daemon 2023-12-01T17:03:09.524 DEBUG:teuthology.orchestra.run.smithi067:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-12-01T17:03:09.526 INFO:tasks.ceph.osd.3:Started 2023-12-01T17:03:09.526 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.527 INFO:tasks.ceph.osd.4:Restarting daemon 2023-12-01T17:03:09.527 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-12-01T17:03:09.529 INFO:tasks.ceph.osd.4:Started 2023-12-01T17:03:09.529 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.529 INFO:tasks.ceph.osd.5:Restarting daemon 2023-12-01T17:03:09.529 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-12-01T17:03:09.531 INFO:tasks.ceph.osd.5:Started 2023-12-01T17:03:09.531 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.531 INFO:tasks.ceph.osd.6:Restarting daemon 2023-12-01T17:03:09.532 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-12-01T17:03:09.533 INFO:tasks.ceph.osd.6:Started 2023-12-01T17:03:09.533 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-12-01T17:03:09.534 INFO:tasks.ceph.osd.7:Restarting daemon 2023-12-01T17:03:09.534 DEBUG:teuthology.orchestra.run.smithi092:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-12-01T17:03:09.536 INFO:tasks.ceph.osd.7:Started 2023-12-01T17:03:09.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:09.590 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:09.589+0000 7fabd1a2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-01T17:03:09.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.664+0000 7f255f597640 10 monclient: get_monmap_and_config 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.664+0000 7f255f597640 10 monclient: build_initial_monmap 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.664+0000 7f255f597640 10 monclient: monmap: 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:03:09.665369+0000 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:03:09.665369+0000 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:09.665 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:09.666 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.665+0000 7f255f597640 1 Processor -- start 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.665+0000 7f255f597640 1 -- start start 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.665+0000 7f255f597640 10 monclient: init 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25580a8a90 0x7f25580b2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient(hunting): picked mon.noname-a con 0x7f25580a8a90 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f25580a8e70 0x7f2558145d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient(hunting): picked mon.noname-c con 0x7f25580a8e70 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2558146290 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25580b1c50 con 0x7f2558146290 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25580b1f30 con 0x7f25580a8a90 2023-12-01T17:03:09.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25580aaa00 con 0x7f25580a8e70 2023-12-01T17:03:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255f597640 10 monclient(hunting): authenticate will time out at 1320.903568s 2023-12-01T17:03:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.666+0000 7f255d333640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25580a8a90 0x7f25580b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.667+0000 7f255d333640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f25580a8a90 0x7f25580b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39966/0 (socket says 172.21.15.67:39966) 2023-12-01T17:03:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.667+0000 7f255d333640 1 -- 172.21.15.67:0/1790105494 learned_addr learned my addr 172.21.15.67:0/1790105494 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:09.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.667+0000 7f255d333640 10 monclient(hunting): get_auth_request con 0x7f25580a8a90 auth_method 0 2023-12-01T17:03:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.667+0000 7f255d333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.667+0000 7f255d333640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.667+0000 7f255d333640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.668+0000 7f255d333640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.668+0000 7f255d333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:09.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.668+0000 7f255d333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.669+0000 7f255d333640 10 monclient(hunting): handle_auth_done global_id 4155 payload 274 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.669+0000 7f255d333640 1 -- 172.21.15.67:0/1790105494 >> v2:172.21.15.92:3300/0 conn(0x7f25580a8e70 msgr2=0x7f2558145d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.669+0000 7f255d333640 1 --2- 172.21.15.67:0/1790105494 >> v2:172.21.15.92:3300/0 conn(0x7f25580a8e70 0x7f2558145d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.669+0000 7f255d333640 1 -- 172.21.15.67:0/1790105494 >> v1:172.21.15.92:6789/0 conn(0x7f2558146290 legacy=0x7f255813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.669+0000 7f255d333640 10 monclient: _finish_hunting 0 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.669+0000 7f255d333640 1 monclient: found mon.noname-a 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.670+0000 7f255d333640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:09.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.670+0000 7f255d333640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:03:09.671 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.670+0000 7f255d333640 1 -- 172.21.15.67:0/1790105494 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25580aace0 con 0x7f25580a8a90 2023-12-01T17:03:09.672 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.672+0000 7f255d333640 1 --2- 172.21.15.67:0/1790105494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25580a8a90 0x7f25580b2250 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f255806bc40 tx=0x7f255402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c5ff00d0bbb0ed server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:09.673 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:09.673+0000 7f4645e3d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-01T17:03:09.676 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f254ffff640 1 -- 172.21.15.67:0/1790105494 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554031750 con 0x7f25580a8a90 2023-12-01T17:03:09.676 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f254ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.676 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f254ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f254ffff640 10 monclient: dump: 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.677 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.678 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f254ffff640 10 monclient: _finish_auth 0 2023-12-01T17:03:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f254ffff640 10 monclient: _check_auth_tickets 2023-12-01T17:03:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f254ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f255f597640 5 monclient: authenticate success, global_id 4155 2023-12-01T17:03:09.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.676+0000 7f255f597640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f254ffff640 1 -- 172.21.15.67:0/1790105494 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25540318b0 con 0x7f25580a8a90 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f254ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f254ffff640 1 -- 172.21.15.67:0/1790105494 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554031c40 con 0x7f25580a8a90 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f255f597640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f255f597640 4 set_mon_vals no callback set 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f254ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f254ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.677+0000 7f254ffff640 10 monclient: dump: 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:09.679 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 10 monclient: shutdown 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 -- 172.21.15.67:0/1790105494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25580a8a90 msgr2=0x7f25580b2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 --2- 172.21.15.67:0/1790105494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25580a8a90 0x7f25580b2250 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f255806bc40 tx=0x7f255402b7f0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 -- 172.21.15.67:0/1790105494 shutdown_connections 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 --2- 172.21.15.67:0/1790105494 >> v2:172.21.15.92:3300/0 conn(0x7f25580a8e70 0x7f2558145d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 --2- 172.21.15.67:0/1790105494 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f25580a8a90 0x7f25580b2250 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 -- 172.21.15.67:0/1790105494 >> 172.21.15.67:0/1790105494 conn(0x7f25580b78d0 msgr2=0x7f25580b7ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 -- 172.21.15.67:0/1790105494 shutdown_connections 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.678+0000 7f255f597640 1 -- 172.21.15.67:0/1790105494 wait complete. 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 10 monclient: build_initial_monmap 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 10 monclient: monmap: 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:09.681 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:09.682 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 1 librados: starting msgr at 2023-12-01T17:03:09.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 1 librados: starting objecter 2023-12-01T17:03:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 1 Processor -- start 2023-12-01T17:03:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 1 -- start start 2023-12-01T17:03:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 1 librados: setting wanted keys 2023-12-01T17:03:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 1 librados: calling monclient init 2023-12-01T17:03:09.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.679+0000 7f255f597640 10 monclient: init 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.680+0000 7f255f597640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.680+0000 7f255f597640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.680+0000 7f255f597640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25580a8a90 0x7f2558144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.680+0000 7f255f597640 10 monclient(hunting): picked mon.noname-b con 0x7f25580a8a90 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.680+0000 7f255f597640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f25580a8e70 0x7f25581435d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 10 monclient(hunting): picked mon.noname-c con 0x7f25580a8e70 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2558146290 0x7f2558131730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 10 monclient(hunting): picked mon.noname-a con 0x7f2558146290 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255cb32640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f25580a8e70 0x7f25581435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25581443f0 con 0x7f2558146290 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2558145420 con 0x7f25580a8a90 2023-12-01T17:03:09.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2558145700 con 0x7f25580a8e70 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255cb32640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f25580a8e70 0x7f25581435d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:38706/0 (socket says 172.21.15.67:38706) 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255cb32640 1 -- 172.21.15.67:0/2101301174 learned_addr learned my addr 172.21.15.67:0/2101301174 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255f597640 10 monclient(hunting): authenticate will time out at 1320.920275s 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255cb32640 10 monclient(hunting): get_auth_request con 0x7f25580a8e70 auth_method 0 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255cb32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255cb32640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.682+0000 7f255cb32640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.684+0000 7f255cb32640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.684+0000 7f255cb32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:09.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.684+0000 7f255cb32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.685+0000 7f255d333640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25580a8a90 0x7f2558144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.685+0000 7f255d333640 10 monclient(hunting): get_auth_request con 0x7f25580a8a90 auth_method 0 2023-12-01T17:03:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.685+0000 7f255d333640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.685+0000 7f255d333640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.685+0000 7f255d333640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.686+0000 7f255d333640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:09.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.686+0000 7f255d333640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.686+0000 7f255d333640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 10 monclient(hunting): handle_auth_done global_id 4142 payload 824 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 1 -- 172.21.15.67:0/2101301174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25580a8a90 msgr2=0x7f2558144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25580a8a90 0x7f2558144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 1 -- 172.21.15.67:0/2101301174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2558146290 msgr2=0x7f2558131730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2558146290 0x7f2558131730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 10 monclient: _finish_hunting 0 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 1 monclient: found mon.noname-c 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:03:09.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.688+0000 7f255cb32640 1 -- 172.21.15.67:0/2101301174 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25540094d0 con 0x7f25580a8e70 2023-12-01T17:03:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.692+0000 7f255cb32640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f25580a8e70 0x7f25581435d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f254800b4a0 tx=0x7f254800b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.692+0000 7f254dffb640 10 client.?.objecter ms_handle_connect 0x7f25580a8e70 2023-12-01T17:03:09.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.692+0000 7f254dffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548010610 con 0x7f25580a8e70 2023-12-01T17:03:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: dump: 2023-12-01T17:03:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:09.701 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: _finish_auth 0 2023-12-01T17:03:09.702 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: _check_auth_tickets 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2548010770 con 0x7f25580a8e70 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548010b20 con 0x7f25580a8e70 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254dffb640 10 monclient: dump: 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:09.703 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.699+0000 7f254f7fe640 4 set_mon_vals no callback set 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.700+0000 7f255f597640 5 monclient: authenticate success, global_id 4142 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.700+0000 7f255f597640 10 monclient: _renew_subs 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.700+0000 7f255f597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.700+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2558143ed0 con 0x7f25580a8e70 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.700+0000 7f255f597640 10 client.4142.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.700+0000 7f255f597640 10 monclient: _renew_subs 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.701+0000 7f255f597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.701+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2558140240 con 0x7f25580a8e70 2023-12-01T17:03:09.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.701+0000 7f255f597640 1 librados: init done 2023-12-01T17:03:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.702+0000 7f255f597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.702+0000 7f255f597640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.702+0000 7f255f597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:09.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.702+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f255804e840 con 0x7f25580a8e70 2023-12-01T17:03:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.707+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2548023480 con 0x7f25580a8e70 2023-12-01T17:03:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.707+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2548022ab0 con 0x7f25580a8e70 2023-12-01T17:03:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.707+0000 7f254dffb640 10 client.4142.objecter ms_dispatch 0x7f2558000b90 osd_map(10..10 src has 1..10) v4 2023-12-01T17:03:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.707+0000 7f254dffb640 3 client.4142.objecter handle_osd_map got epochs [10,10] > 0 2023-12-01T17:03:09.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.707+0000 7f254dffb640 3 client.4142.objecter handle_osd_map decoding full epoch 10 2023-12-01T17:03:09.708 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.707+0000 7f254dffb640 20 client.4142.objecter dump_active .. 0 homeless 2023-12-01T17:03:09.761 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:09.761+0000 7f4645e3d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-01T17:03:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.878+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f254801e060 con 0x7f25580a8e70 2023-12-01T17:03:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.879+0000 7f254dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:09.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:09.879+0000 7f254dffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:09.941 INFO:tasks.ceph.mgr.x.smithi092.stderr:2023-12-01T17:03:09.941+0000 7f4645e3d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-01T17:03:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.001+0000 7f255f597640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.002+0000 7f255f597640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.002+0000 7f255f597640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:10.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.002+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25580726a0 con 0x7f25580a8e70 2023-12-01T17:03:10.179 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.179+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mgrmap(e 2) v1 ==== 44567+0+0 (secure 0 0 0) 0x7f25480223e0 con 0x7f25580a8e70 2023-12-01T17:03:10.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.282+0000 7f254dffb640 1 -- 172.21.15.67:0/2101301174 <== mon.2 v2:172.21.15.67:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f254800d300 con 0x7f25580a8e70 2023-12-01T17:03:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.282+0000 7f254dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:10.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.282+0000 7f254dffb640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:10.283 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:03:10.283 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:09.418337+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:10.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 10 librados: watch_flush enter 2023-12-01T17:03:10.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 10 librados: watch_flush exit 2023-12-01T17:03:10.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 20 client.4142.objecter shutdown clearing up homeless session... 2023-12-01T17:03:10.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 10 client.4142.objecter successfully canceled tick 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 10 monclient: shutdown 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f25580a8e70 msgr2=0x7f25581435d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f25580a8e70 0x7f25581435d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f254800b4a0 tx=0x7f254800b960 comp rx=0 tx=0).stop 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 shutdown_connections 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2558146290 0x7f2558131730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f25580a8e70 0x7f25581435d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 --2- 172.21.15.67:0/2101301174 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f25580a8a90 0x7f2558144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 >> 172.21.15.67:0/2101301174 conn(0x7f25580b78d0 msgr2=0x7f25580b7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 shutdown_connections 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.283+0000 7f255f597640 1 -- 172.21.15.67:0/2101301174 wait complete. 2023-12-01T17:03:10.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:10.284+0000 7f255f597640 1 librados: shutdown 2023-12-01T17:03:10.299 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-12-01T17:03:10.299 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-01T17:03:11.106 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.266+0000 7f940e890640 10 monclient: get_monmap_and_config 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.266+0000 7f940e890640 10 monclient: build_initial_monmap 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.266+0000 7f940e890640 10 monclient: monmap: 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:03:11.267107+0000 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:03:11.267107+0000 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:11.267 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:11.268 INFO:teuthology.misc.health.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:11.268 INFO:teuthology.misc.health.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:11.268 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:11.268 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.267+0000 7f940e890640 1 Processor -- start 2023-12-01T17:03:11.268 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.267+0000 7f940e890640 1 -- start start 2023-12-01T17:03:11.268 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.267+0000 7f940e890640 10 monclient: init 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f94080a8900 0x7f940813efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient(hunting): picked mon.noname-a con 0x7f94080a8900 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94080a9b80 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f94080a8ce0 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94080b1b80 con 0x7f94080a9b80 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94080aa870 con 0x7f94080a8ce0 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94080aab50 con 0x7f94080a8900 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:11.269 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.268+0000 7f940e890640 10 monclient(hunting): authenticate will time out at 1322.505977s 2023-12-01T17:03:11.364 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.364+0000 7f940ce2d640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f94080a8ce0 0x7f94081453f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:34678/0 (socket says 172.21.15.67:34678) 2023-12-01T17:03:11.365 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.364+0000 7f940ce2d640 1 -- 172.21.15.67:0/252424737 learned_addr learned my addr 172.21.15.67:0/252424737 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:03:11.375 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.375+0000 7f9407fff640 1 --2- 172.21.15.67:0/252424737 >> v2:172.21.15.67:3300/0 conn(0x7f94080a8900 0x7f940813efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:11.376 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.376+0000 7f9407fff640 10 monclient(hunting): get_auth_request con 0x7f94080a8900 auth_method 0 2023-12-01T17:03:11.376 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.376+0000 7f9407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:11.376 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.376+0000 7f9407fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:11.376 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.376+0000 7f9407fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:11.392 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.392+0000 7f9407fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:11.392 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.392+0000 7f9407fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:11.392 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.392+0000 7f9407fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 10 monclient(hunting): handle_auth_done global_id 4158 payload 274 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 1 -- 172.21.15.67:0/252424737 >> v1:172.21.15.92:6789/0 conn(0x7f94080a8ce0 legacy=0x7f94081453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 1 -- 172.21.15.67:0/252424737 >> v1:172.21.15.67:6789/0 conn(0x7f94080a9b80 legacy=0x7f940813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 1 monclient: found mon.noname-a 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:03:11.399 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.398+0000 7f9407fff640 1 -- 172.21.15.67:0/252424737 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9408145b00 con 0x7f94080a8900 2023-12-01T17:03:11.417 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:11.417+0000 7fab72622640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-12-01T17:03:11.434 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.433+0000 7f9407fff640 1 --2- 172.21.15.67:0/252424737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8900 0x7f940813efd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f93f80097a0 tx=0x7f93f802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3142a95b5d73d9 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:11.438 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:03:11.438+0000 7fab72622640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-12-01T17:03:11.457 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 1 -- 172.21.15.67:0/252424737 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8030860 con 0x7f94080a8900 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: dump: 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:11.458 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: _finish_auth 0 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 1 -- 172.21.15.67:0/252424737 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f80309c0 con 0x7f94080a8900 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 1 -- 172.21.15.67:0/252424737 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f8030d90 con 0x7f94080a8900 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f9406ffd640 10 monclient: dump: 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:11.459 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f940e890640 5 monclient: authenticate success, global_id 4158 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f940e890640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.456+0000 7f940e890640 4 set_mon_vals no callback set 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 10 monclient: shutdown 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 1 -- 172.21.15.67:0/252424737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8900 msgr2=0x7f940813efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 1 --2- 172.21.15.67:0/252424737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8900 0x7f940813efd0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f93f80097a0 tx=0x7f93f802b9a0 comp rx=0 tx=0).stop 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 1 -- 172.21.15.67:0/252424737 shutdown_connections 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 1 --2- 172.21.15.67:0/252424737 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8900 0x7f940813efd0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 1 -- 172.21.15.67:0/252424737 >> 172.21.15.67:0/252424737 conn(0x7f9408066f00 msgr2=0x7f94080b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:11.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 1 -- 172.21.15.67:0/252424737 shutdown_connections 2023-12-01T17:03:11.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.457+0000 7f940e890640 1 -- 172.21.15.67:0/252424737 wait complete. 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.458+0000 7f940e890640 10 monclient: build_initial_monmap 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.458+0000 7f940e890640 10 monclient: monmap: 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:11.462 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.459+0000 7f940e890640 1 librados: starting msgr at 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.459+0000 7f940e890640 1 librados: starting objecter 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.459+0000 7f940e890640 1 Processor -- start 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.459+0000 7f940e890640 1 -- start start 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.459+0000 7f940e890640 1 librados: setting wanted keys 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.459+0000 7f940e890640 1 librados: calling monclient init 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.459+0000 7f940e890640 10 monclient: init 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f94080a8900 0x7f94081dae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): picked mon.noname-c con 0x7f94080a8900 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8ce0 0x7f94081db340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): picked mon.noname-a con 0x7f94080a8ce0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94080a9b80 0x7f940812e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): picked mon.noname-b con 0x7f94080a9b80 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:11.463 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94081411f0 con 0x7f94080a8ce0 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94081400b0 con 0x7f94080a9b80 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9408140220 con 0x7f94080a8900 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.460+0000 7f940e890640 10 monclient(hunting): authenticate will time out at 1322.698374s 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.461+0000 7f9407fff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f94080a8900 0x7f94081dae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:11.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.461+0000 7f940ce2d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94080a9b80 0x7f940812e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:11.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f940ce2d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94080a9b80 0x7f940812e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42640/0 (socket says 172.21.15.67:42640) 2023-12-01T17:03:11.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f940ce2d640 1 -- 172.21.15.67:0/3248227212 learned_addr learned my addr 172.21.15.67:0/3248227212 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:11.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f940ce2d640 10 monclient(hunting): get_auth_request con 0x7f94080a9b80 auth_method 0 2023-12-01T17:03:11.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f940ce2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:11.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f940ce2d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:11.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f940ce2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f9407fff640 10 monclient(hunting): get_auth_request con 0x7f94080a8900 auth_method 0 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f9407fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f9407fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.462+0000 7f9407fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.463+0000 7f940ce2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.463+0000 7f940ce2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.463+0000 7f940ce2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.463+0000 7f9407fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.463+0000 7f9407fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:11.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.463+0000 7f9407fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 10 monclient(hunting): handle_auth_done global_id 4183 payload 824 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 1 -- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f94080a8900 msgr2=0x7f94081dae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f94080a8900 0x7f94081dae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 1 -- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8ce0 msgr2=0x7f94081db340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8ce0 0x7f94081db340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 10 monclient: _finish_hunting 0 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 1 monclient: found mon.noname-b 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:11.468 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.467+0000 7f940ce2d640 1 -- 172.21.15.67:0/3248227212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f8009400 con 0x7f94080a9b80 2023-12-01T17:03:11.471 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.471+0000 7f940ce2d640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94080a9b80 0x7f940812e430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f940000cbc0 tx=0x7f940000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:11.471 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.471+0000 7f9404ff9640 10 client.?.objecter ms_handle_connect 0x7f94080a9b80 2023-12-01T17:03:11.471 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.471+0000 7f9404ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 1 -- 172.21.15.67:0/3248227212 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000ad40 con 0x7f94080a9b80 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 10 monclient: dump: 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:11.477 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 10 monclient: _finish_auth 0 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 1 -- 172.21.15.67:0/3248227212 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9400012400 con 0x7f94080a9b80 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f940e890640 5 monclient: authenticate success, global_id 4183 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f940e890640 10 monclient: _renew_subs 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f940e890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f940813fc20 con 0x7f94080a9b80 2023-12-01T17:03:11.478 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f9404ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:11.479 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f940e890640 10 client.4183.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:11.479 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.476+0000 7f94067fc640 4 set_mon_vals no callback set 2023-12-01T17:03:11.479 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f940e890640 10 monclient: _renew_subs 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f940e890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94081dba60 con 0x7f94080a9b80 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f940e890640 1 librados: init done 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f9404ff9640 1 -- 172.21.15.67:0/3248227212 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f940000aea0 con 0x7f94080a9b80 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f9404ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f9404ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.477+0000 7f9404ff9640 10 monclient: dump: 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:11.480 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:11.481 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:11.481 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:11.481 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.479+0000 7f940e890640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:11.481 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.479+0000 7f940e890640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:11.481 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.479+0000 7f940e890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:11.481 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.479+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9408144b60 con 0x7f94080a9b80 2023-12-01T17:03:11.511 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.511+0000 7f9404ff9640 1 -- 172.21.15.67:0/3248227212 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 3) v1 ==== 89069+0+0 (secure 0 0 0) 0x7f9400007070 con 0x7f94080a9b80 2023-12-01T17:03:11.512 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.512+0000 7f9404ff9640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f93f406c6a0 0x7f93f406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:11.512 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.512+0000 7f9407fff640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f93f406c6a0 0x7f93f406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:11.512 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.512+0000 7f9407fff640 10 monclient: get_auth_request con 0x7f93f406c6a0 auth_method 0 2023-12-01T17:03:11.512 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.512+0000 7f9407fff640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f93f406c6a0 0x7f93f406eb60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f93f8036000 tx=0x7f93f8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:11.512 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.512+0000 7f9404ff9640 10 client.4183.objecter ms_handle_connect 0x7f93f406c6a0 2023-12-01T17:03:11.946 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.945+0000 7f9404ff9640 1 -- 172.21.15.67:0/3248227212 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f94000584d0 con 0x7f94080a9b80 2023-12-01T17:03:11.946 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.945+0000 7f9404ff9640 10 client.4183.objecter ms_dispatch 0x7f9408000b90 osd_map(10..10 src has 1..10) v4 2023-12-01T17:03:11.946 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.945+0000 7f9404ff9640 3 client.4183.objecter handle_osd_map got epochs [10,10] > 0 2023-12-01T17:03:11.946 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.945+0000 7f9404ff9640 3 client.4183.objecter handle_osd_map decoding full epoch 10 2023-12-01T17:03:11.946 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.945+0000 7f9404ff9640 20 client.4183.objecter dump_active .. 0 homeless 2023-12-01T17:03:11.947 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.947+0000 7f9404ff9640 1 -- 172.21.15.67:0/3248227212 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9400054810 con 0x7f94080a9b80 2023-12-01T17:03:11.947 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.947+0000 7f9404ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:11.947 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:11.947+0000 7f9404ff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:12.314 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.314+0000 7f940e890640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:12.314 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.314+0000 7f940e890640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:12.314 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.314+0000 7f940e890640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:12.315 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.314+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9408144b60 con 0x7f94080a9b80 2023-12-01T17:03:12.504 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.504+0000 7f9404ff9640 1 -- 172.21.15.67:0/3248227212 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9408144b60 con 0x7f94080a9b80 2023-12-01T17:03:12.504 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.504+0000 7f9404ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:12.504 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.504+0000 7f9404ff9640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:12.505 INFO:teuthology.misc.health.smithi067.stdout: 2023-12-01T17:03:12.505 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:09.418337+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 10 librados: watch_flush enter 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 10 librados: watch_flush exit 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 20 client.4183.objecter shutdown clearing up homeless session... 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 10 client.4183.objecter successfully canceled tick 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f93f406c6a0 msgr2=0x7f93f406eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f93f406c6a0 0x7f93f406eb60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f93f8036000 tx=0x7f93f8008030 comp rx=0 tx=0).stop 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 10 monclient: shutdown 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94080a9b80 msgr2=0x7f940812e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94080a9b80 0x7f940812e430 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f940000cbc0 tx=0x7f940000a430 comp rx=0 tx=0).stop 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 shutdown_connections 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f94080a9b80 0x7f940812e430 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:12.508 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f94080a8ce0 0x7f94081db340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:12.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f93f406c6a0 0x7f93f406eb60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:12.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 --2- 172.21.15.67:0/3248227212 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f94080a8900 0x7f94081dae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:12.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 >> 172.21.15.67:0/3248227212 conn(0x7f9408066f00 msgr2=0x7f940813eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:12.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.507+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 shutdown_connections 2023-12-01T17:03:12.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.508+0000 7f940e890640 1 -- 172.21.15.67:0/3248227212 wait complete. 2023-12-01T17:03:12.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:12.508+0000 7f940e890640 1 librados: shutdown 2023-12-01T17:03:12.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-01T17:03:17.440 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:07.843 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:17.517 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:07.919 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:17.596 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:07.978 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:17.654 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:08.039 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:17.842 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:08.234 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:17.913 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:08.306 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.061 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:03:18.054+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:18.102 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:08.504 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.167 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:08.569 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.217 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-12-01T17:03:18.211+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:18.400 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:08.780 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.420 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:08.823 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.461 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:08.842 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.485 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:08.887 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.490 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:08.894 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.498 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:03:18.491+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:18.673 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:09.058 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.704 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:03:18.697+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:18.809 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:09.197 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.843 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:09.245 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.877 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:09.265 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.941 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:09.333 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:18.993 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:09.380 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.016 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:03:19.009+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:19.029 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:09.414 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.039 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:03:19.030+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:19.058 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:09.444 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.148 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:09.550 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.158 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:09.560 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.295 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:09.687 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.338 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:19.347 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:09.732 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.426 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:09.829 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.433 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.432+0000 7f3e2ab41640 10 monclient: get_monmap_and_config 2023-12-01T17:03:19.433 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.432+0000 7f3e2ab41640 10 monclient: build_initial_monmap 2023-12-01T17:03:19.433 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.432+0000 7f3e2ab41640 10 monclient: monmap: 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:03:19.434273+0000 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:03:19.434273+0000 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:19.434 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:19.435 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.433+0000 7f3e2ab41640 1 Processor -- start 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.433+0000 7f3e2ab41640 1 -- start start 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.433+0000 7f3e2ab41640 10 monclient: init 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.433+0000 7f3e2ab41640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.433+0000 7f3e2ab41640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.433+0000 7f3e2ab41640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8890 0x7f3e240b2020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.433+0000 7f3e2ab41640 10 monclient(hunting): picked mon.noname-b con 0x7f3e240a8890 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e240a8c70 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e2413f0a0 0x7f3e2413f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 10 monclient(hunting): picked mon.noname-a con 0x7f3e2413f0a0 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e2413fa70 con 0x7f3e240a8c70 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e240b2560 con 0x7f3e2413f0a0 2023-12-01T17:03:19.436 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:19.437 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240a9c30 con 0x7f3e240a8890 2023-12-01T17:03:19.437 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:19.437 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e2ab41640 10 monclient(hunting): authenticate will time out at 1330.672467s 2023-12-01T17:03:19.437 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e290de640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e2413f0a0 0x7f3e2413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:19.437 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.434+0000 7f3e288dd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8890 0x7f3e240b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e288dd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8890 0x7f3e240b2020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:44306/0 (socket says 172.21.15.67:44306) 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e288dd640 1 -- 172.21.15.67:0/48184868 learned_addr learned my addr 172.21.15.67:0/48184868 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e288dd640 10 monclient(hunting): get_auth_request con 0x7f3e240a8890 auth_method 0 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e288dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e288dd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e288dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e290de640 10 monclient(hunting): get_auth_request con 0x7f3e2413f0a0 auth_method 0 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e290de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e290de640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:19.438 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.435+0000 7f3e290de640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:19.439 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.436+0000 7f3e288dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:19.439 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.436+0000 7f3e288dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:19.439 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.436+0000 7f3e288dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:19.439 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.436+0000 7f3e290de640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:19.439 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.436+0000 7f3e290de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:19.439 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.436+0000 7f3e290de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:19.439 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 10 monclient(hunting): handle_auth_done global_id 4172 payload 274 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 1 -- 172.21.15.67:0/48184868 >> v2:172.21.15.67:3300/0 conn(0x7f3e2413f0a0 msgr2=0x7f3e2413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 1 --2- 172.21.15.67:0/48184868 >> v2:172.21.15.67:3300/0 conn(0x7f3e2413f0a0 0x7f3e2413f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 1 -- 172.21.15.67:0/48184868 >> v1:172.21.15.67:6789/0 conn(0x7f3e240a8c70 legacy=0x7f3e24142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 10 monclient: _finish_hunting 0 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 1 monclient: found mon.noname-b 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:03:19.440 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.438+0000 7f3e288dd640 1 -- 172.21.15.67:0/48184868 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240aada0 con 0x7f3e240a8890 2023-12-01T17:03:19.442 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.441+0000 7f3e288dd640 1 --2- 172.21.15.67:0/48184868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8890 0x7f3e240b2020 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009870 tx=0x7f3e1802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4e33e8692b1ad1d server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:19.448 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 1 -- 172.21.15.67:0/48184868 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18030840 con 0x7f3e240a8890 2023-12-01T17:03:19.448 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:19.448 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: dump: 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:19.449 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: _finish_auth 0 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: _check_auth_tickets 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 1 -- 172.21.15.67:0/48184868 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e180309a0 con 0x7f3e240a8890 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 5 monclient: authenticate success, global_id 4172 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 4 set_mon_vals no callback set 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 1 -- 172.21.15.67:0/48184868 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18030d30 con 0x7f3e240a8890 2023-12-01T17:03:19.450 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e237fe640 10 monclient: dump: 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:19.451 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 10 monclient: shutdown 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 1 -- 172.21.15.67:0/48184868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8890 msgr2=0x7f3e240b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/48184868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8890 0x7f3e240b2020 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009870 tx=0x7f3e1802b980 comp rx=0 tx=0).stop 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.446+0000 7f3e2ab41640 1 -- 172.21.15.67:0/48184868 shutdown_connections 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.447+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/48184868 >> v2:172.21.15.67:3300/0 conn(0x7f3e2413f0a0 0x7f3e2413f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.447+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/48184868 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8890 0x7f3e240b2020 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.447+0000 7f3e2ab41640 1 -- 172.21.15.67:0/48184868 >> 172.21.15.67:0/48184868 conn(0x7f3e24066db0 msgr2=0x7f3e240b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.447+0000 7f3e2ab41640 1 -- 172.21.15.67:0/48184868 shutdown_connections 2023-12-01T17:03:19.452 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.447+0000 7f3e2ab41640 1 -- 172.21.15.67:0/48184868 wait complete. 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 10 monclient: build_initial_monmap 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 10 monclient: monmap: 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 1 librados: starting msgr at 2023-12-01T17:03:19.454 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 1 librados: starting objecter 2023-12-01T17:03:19.455 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 1 Processor -- start 2023-12-01T17:03:19.455 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 1 -- start start 2023-12-01T17:03:19.455 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 1 librados: setting wanted keys 2023-12-01T17:03:19.455 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 1 librados: calling monclient init 2023-12-01T17:03:19.455 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.448+0000 7f3e2ab41640 10 monclient: init 2023-12-01T17:03:19.455 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:09.831 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.458 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:19.458 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:03:19.458 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240a8890 0x7f3e24145560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): picked mon.noname-b con 0x7f3e240a8890 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8c70 0x7f3e241dad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): picked mon.noname-c con 0x7f3e240a8c70 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2413f0a0 0x7f3e241db2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): picked mon.noname-a con 0x7f3e2413f0a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2413c100 con 0x7f3e2413f0a0 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2413a350 con 0x7f3e240a8890 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2413c350 con 0x7f3e240a8c70 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.449+0000 7f3e2ab41640 10 monclient(hunting): authenticate will time out at 1330.688378s 2023-12-01T17:03:19.459 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.450+0000 7f3e23fff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8c70 0x7f3e241dad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.450+0000 7f3e288dd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240a8890 0x7f3e24145560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.450+0000 7f3e290de640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2413f0a0 0x7f3e241db2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8c70 0x7f3e241dad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:44308/0 (socket says 172.21.15.67:44308) 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240a8890 0x7f3e24145560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:48266/0 (socket says 172.21.15.67:48266) 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 1 -- 172.21.15.67:0/2608172361 learned_addr learned my addr 172.21.15.67:0/2608172361 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 10 monclient(hunting): get_auth_request con 0x7f3e240a8c70 auth_method 0 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 10 monclient(hunting): get_auth_request con 0x7f3e240a8890 auth_method 0 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:19.460 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e290de640 10 monclient(hunting): get_auth_request con 0x7f3e2413f0a0 auth_method 0 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e290de640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e290de640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e290de640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e288dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e290de640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e290de640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.451+0000 7f3e290de640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 10 monclient(hunting): handle_auth_done global_id 4216 payload 824 2023-12-01T17:03:19.461 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 1 -- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8c70 msgr2=0x7f3e241dad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8c70 0x7f3e241dad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 1 -- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2413f0a0 msgr2=0x7f3e241db2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2413f0a0 0x7f3e241db2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 10 monclient: _finish_hunting 0 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 1 monclient: found mon.noname-b 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.454+0000 7f3e288dd640 1 -- 172.21.15.67:0/2608172361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e180094d0 con 0x7f3e240a8890 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.456+0000 7f3e288dd640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240a8890 0x7f3e24145560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e18033000 tx=0x7f3e18004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.456+0000 7f3e217fa640 10 client.?.objecter ms_handle_connect 0x7f3e240a8890 2023-12-01T17:03:19.462 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.456+0000 7f3e217fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:19.463 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:03:19.451+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:19.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 1 -- 172.21.15.67:0/2608172361 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18039c10 con 0x7f3e240a8890 2023-12-01T17:03:19.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:19.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:19.464 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: dump: 2023-12-01T17:03:19.464 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: _finish_auth 0 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: _check_auth_tickets 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 1 -- 172.21.15.67:0/2608172361 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e18039d70 con 0x7f3e240a8890 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:19.465 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 1 -- 172.21.15.67:0/2608172361 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18043640 con 0x7f3e240a8890 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e217fa640 10 monclient: dump: 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:19.466 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 5 monclient: authenticate success, global_id 4216 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e227fc640 4 set_mon_vals no callback set 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 10 monclient: _renew_subs 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2413c630 con 0x7f3e240a8890 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 10 monclient: _renew_subs 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e241db800 con 0x7f3e240a8890 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.460+0000 7f3e2ab41640 1 librados: init done 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.461+0000 7f3e2ab41640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.461+0000 7f3e2ab41640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.461+0000 7f3e2ab41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:19.467 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.461+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2404e840 con 0x7f3e240a8890 2023-12-01T17:03:19.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.479+0000 7f3e217fa640 1 -- 172.21.15.67:0/2608172361 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f3e180437a0 con 0x7f3e240a8890 2023-12-01T17:03:19.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.479+0000 7f3e217fa640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e0806c7e0 0x7f3e0806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:19.480 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.479+0000 7f3e23fff640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e0806c7e0 0x7f3e0806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:19.481 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.479+0000 7f3e23fff640 10 monclient: get_auth_request con 0x7f3e0806c7e0 auth_method 0 2023-12-01T17:03:19.481 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.480+0000 7f3e23fff640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e0806c7e0 0x7f3e0806eca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3e24072a00 tx=0x7f3e10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:19.481 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.480+0000 7f3e217fa640 10 client.4216.objecter ms_handle_connect 0x7f3e0806c7e0 2023-12-01T17:03:19.483 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.481+0000 7f3e217fa640 1 -- 172.21.15.67:0/2608172361 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f3e1803e070 con 0x7f3e240a8890 2023-12-01T17:03:19.483 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.481+0000 7f3e217fa640 10 client.4216.objecter ms_dispatch 0x7f3e24000b90 osd_map(10..10 src has 1..10) v4 2023-12-01T17:03:19.483 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.481+0000 7f3e217fa640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2023-12-01T17:03:19.483 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.481+0000 7f3e217fa640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2023-12-01T17:03:19.483 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.481+0000 7f3e217fa640 20 client.4216.objecter dump_active .. 0 homeless 2023-12-01T17:03:19.488 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:09.892 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.509 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:09.911 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.616 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.009 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.618 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.003 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.627 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:03:19.616+0000 5dc2340 -1 Falling back to public interface 2023-12-01T17:03:19.712 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.115 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.719 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.717+0000 7f3e217fa640 1 -- 172.21.15.67:0/2608172361 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e18078060 con 0x7f3e240a8890 2023-12-01T17:03:19.719 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.718+0000 7f3e217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:19.719 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.718+0000 7f3e217fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:19.811 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:10.191 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.822 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:10.224 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.890 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:10.293 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.896 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.280 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.896 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.287 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.970 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:10.358 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.973 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.971+0000 7f3e2ab41640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:19.973 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.971+0000 7f3e2ab41640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:19.973 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.971+0000 7f3e2ab41640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:19.973 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.971+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e2413f850 con 0x7f3e240a8890 2023-12-01T17:03:19.976 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.379 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:19.987 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.986+0000 7f3e217fa640 1 -- 172.21.15.67:0/2608172361 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f3e18034070 con 0x7f3e240a8890 2023-12-01T17:03:19.988 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.986+0000 7f3e217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:19.988 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.986+0000 7f3e217fa640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:19.988 INFO:teuthology.misc.health.smithi067.stdout: 2023-12-01T17:03:19.988 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:09.418337+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:19.990 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 10 librados: watch_flush enter 2023-12-01T17:03:19.990 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 10 librados: watch_flush exit 2023-12-01T17:03:19.990 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 20 client.4216.objecter shutdown clearing up homeless session... 2023-12-01T17:03:19.990 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 10 client.4216.objecter successfully canceled tick 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e0806c7e0 msgr2=0x7f3e0806eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e0806c7e0 0x7f3e0806eca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3e24072a00 tx=0x7f3e10009040 comp rx=0 tx=0).stop 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 10 monclient: shutdown 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240a8890 msgr2=0x7f3e24145560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.988+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240a8890 0x7f3e24145560 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3e18033000 tx=0x7f3e18004730 comp rx=0 tx=0).stop 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 shutdown_connections 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e2413f0a0 0x7f3e241db2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e0806c7e0 0x7f3e0806eca0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3e240a8c70 0x7f3e241dad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 --2- 172.21.15.67:0/2608172361 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3e240a8890 0x7f3e24145560 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:19.991 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 >> 172.21.15.67:0/2608172361 conn(0x7f3e24066db0 msgr2=0x7f3e2413e090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:19.992 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 shutdown_connections 2023-12-01T17:03:19.992 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 -- 172.21.15.67:0/2608172361 wait complete. 2023-12-01T17:03:19.992 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:19.989+0000 7f3e2ab41640 1 librados: shutdown 2023-12-01T17:03:20.003 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-01T17:03:20.105 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:10.507 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.148 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:10.536 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.182 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.574 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.182 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.566 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.183 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:10.563 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.256 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:10.660 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.258 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.660 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.345 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:10.733 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.393 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:10.795 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.474 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:10.866 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.474 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:10.858 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.475 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:10.855 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.503 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:10.891 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.545 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:10.947 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.556 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:10.959 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.681 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:11.083 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.705 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:11.093 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.763 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:11.147 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.765 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:11.158 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.767 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:11.147 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.837 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:11.239 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.851 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:11.254 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.864 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:11.252 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.975 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:11.377 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:20.994 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:11.382 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.053 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:11.437 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.056 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:11.436 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.057 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:11.450 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.140 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:11.543 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.162 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:11.550 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.268 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:11.670 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.287 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:11.675 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.315 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:11.717 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.348 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:11.751 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.349 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:11.729 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.353 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:11.746 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.402 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:11.805 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.435 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:11.838 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.456 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:11.844 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.576 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:11.961 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.583 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:11.971 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.630 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:12.015 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.638 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:12.018 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.725 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:12.127 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.755 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:12.143 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.876 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:12.264 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.890 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:12.292 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.937 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:12.317 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.959 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:12.351 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:21.973 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:12.374 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.014 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:12.406 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.017 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:12.421 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.047 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:12.434 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.166 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:12.553 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.317 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:12.705 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.431 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:12.818 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.454 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:12.834 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.507 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:12.910 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.510 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:12.890 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.563 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:12.967 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.583 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:12.971 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:22.959 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:13.347 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:23.019 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:13.407 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:23.081 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:13.468 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:23.137 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:13.525 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:26.810 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:26.907 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.906+0000 7f12f4b8a640 10 monclient: get_monmap_and_config 2023-12-01T17:03:26.907 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.906+0000 7f12f4b8a640 10 monclient: build_initial_monmap 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.906+0000 7f12f4b8a640 10 monclient: monmap: 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:03:26.907535+0000 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:03:26.907535+0000 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:26.908 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:26.909 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:26.909 INFO:teuthology.misc.health.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:26.909 INFO:teuthology.misc.health.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:26.909 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:26.909 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.907+0000 7f12f4b8a640 1 Processor -- start 2023-12-01T17:03:26.909 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.907+0000 7f12f4b8a640 1 -- start start 2023-12-01T17:03:26.909 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.907+0000 7f12f4b8a640 10 monclient: init 2023-12-01T17:03:26.911 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:26.911 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-12-01T17:03:26.911 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12f00a8890 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f12f00a8c70 0x7f12f0142730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient(hunting): picked mon.noname-c con 0x7f12f00a8c70 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12f013f0a0 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f00a9c30 con 0x7f12f013f0a0 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f00aada0 con 0x7f12f00a8890 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f0142210 con 0x7f12f00a8c70 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.908+0000 7f12f4b8a640 10 monclient(hunting): authenticate will time out at 1338.146265s 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.909+0000 7f12edd9b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f12f00a8c70 0x7f12f0142730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.909+0000 7f12eed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f12f013f0a0 0x7f12f013f510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47854/0 (socket says 172.21.15.67:47854) 2023-12-01T17:03:26.912 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.909+0000 7f12eed9d640 1 -- 172.21.15.67:0/1539343605 learned_addr learned my addr 172.21.15.67:0/1539343605 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:03:26.913 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.910+0000 7f12edd9b640 10 monclient(hunting): get_auth_request con 0x7f12f00a8c70 auth_method 0 2023-12-01T17:03:26.913 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.910+0000 7f12edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:26.913 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.910+0000 7f12edd9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:26.913 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.910+0000 7f12edd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:26.913 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.911+0000 7f12edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:26.913 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.911+0000 7f12edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:26.913 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.911+0000 7f12edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:26.914 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12ed59a640 1 -- 172.21.15.67:0/1539343605 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383620058 0 0) 0x7f12f00a9c30 con 0x7f12f013f0a0 2023-12-01T17:03:26.914 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12ed59a640 10 monclient(hunting): _negotiate 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12ed59a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12ed59a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12ed59a640 10 monclient(hunting): my global_id is 4194 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12ed59a640 1 -- 172.21.15.67:0/1539343605 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12d8003520 con 0x7f12f013f0a0 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 10 monclient(hunting): handle_auth_done global_id 4219 payload 274 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 1 -- 172.21.15.67:0/1539343605 >> v1:172.21.15.92:6789/0 conn(0x7f12f00a8890 legacy=0x7f12f00b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 1 -- 172.21.15.67:0/1539343605 >> v1:172.21.15.67:6789/0 conn(0x7f12f013f0a0 legacy=0x7f12f013f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 10 monclient: _finish_hunting 0 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 1 monclient: found mon.noname-c 2023-12-01T17:03:26.915 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:26.916 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:03:26.916 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.913+0000 7f12edd9b640 1 -- 172.21.15.67:0/1539343605 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f0142c70 con 0x7f12f00a8c70 2023-12-01T17:03:26.920 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.920+0000 7f12edd9b640 1 --2- 172.21.15.67:0/1539343605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8c70 0x7f12f0142730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12e40097a0 tx=0x7f12e402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34d37f55ff4c8a58 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 1 -- 172.21.15.67:0/1539343605 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4030840 con 0x7f12f00a8c70 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: dump: 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.935 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: _finish_auth 0 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: _check_auth_tickets 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 1 -- 172.21.15.67:0/1539343605 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e40309a0 con 0x7f12f00a8c70 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 1 -- 172.21.15.67:0/1539343605 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4030d70 con 0x7f12f00a8c70 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12f4b8a640 5 monclient: authenticate success, global_id 4219 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12f4b8a640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:26.936 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12f4b8a640 4 set_mon_vals no callback set 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.928+0000 7f12ed59a640 10 monclient: dump: 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 10 monclient: shutdown 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:26.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 1 -- 172.21.15.67:0/1539343605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8c70 msgr2=0x7f12f0142730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/1539343605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8c70 0x7f12f0142730 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f12e40097a0 tx=0x7f12e402b980 comp rx=0 tx=0).stop 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 1 -- 172.21.15.67:0/1539343605 shutdown_connections 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/1539343605 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8c70 0x7f12f0142730 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 1 -- 172.21.15.67:0/1539343605 >> 172.21.15.67:0/1539343605 conn(0x7f12f0066db0 msgr2=0x7f12f00b76a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 1 -- 172.21.15.67:0/1539343605 shutdown_connections 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.929+0000 7f12f4b8a640 1 -- 172.21.15.67:0/1539343605 wait complete. 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 10 monclient: build_initial_monmap 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 10 monclient: monmap: 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2023-12-01T17:03:26.938 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 1 librados: starting msgr at 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 1 librados: starting objecter 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 1 Processor -- start 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 1 -- start start 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 1 librados: setting wanted keys 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 1 librados: calling monclient init 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.930+0000 7f12f4b8a640 10 monclient: init 2023-12-01T17:03:26.939 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8890 0x7f12f01377d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 10 monclient(hunting): picked mon.noname-b con 0x7f12f00a8890 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f00a8c70 0x7f12f0139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 10 monclient(hunting): picked mon.noname-c con 0x7f12f00a8c70 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f013f0a0 0x7f12f0139b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.931+0000 7f12f4b8a640 10 monclient(hunting): picked mon.noname-a con 0x7f12f013f0a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0138b00 con 0x7f12f013f0a0 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0138de0 con 0x7f12f00a8890 2023-12-01T17:03:26.940 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12ee59c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8890 0x7f12f01377d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f01390c0 con 0x7f12f00a8c70 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12f4b8a640 10 monclient(hunting): authenticate will time out at 1338.169720s 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12edd9b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f00a8c70 0x7f12f0139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12ee59c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8890 0x7f12f01377d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:51264/0 (socket says 172.21.15.67:51264) 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12ee59c640 1 -- 172.21.15.67:0/624507848 learned_addr learned my addr 172.21.15.67:0/624507848 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12ee59c640 10 monclient(hunting): get_auth_request con 0x7f12f00a8890 auth_method 0 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12ee59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:26.941 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12ee59c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12ee59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12edd9b640 10 monclient(hunting): get_auth_request con 0x7f12f00a8c70 auth_method 0 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12edd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12edd9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.932+0000 7f12edd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.933+0000 7f12edd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.933+0000 7f12edd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.933+0000 7f12edd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.933+0000 7f12ee59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.933+0000 7f12ee59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:26.942 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.933+0000 7f12ee59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:26.944 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12eed9d640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f013f0a0 0x7f12f0139b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:26.944 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 10 monclient(hunting): handle_auth_done global_id 4225 payload 824 2023-12-01T17:03:26.944 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 1 -- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f00a8c70 msgr2=0x7f12f0139650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:26.944 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f00a8c70 0x7f12f0139650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 1 -- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f013f0a0 msgr2=0x7f12f0139b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f013f0a0 0x7f12f0139b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 10 monclient: _finish_hunting 0 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 1 monclient: found mon.noname-b 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.937+0000 7f12ee59c640 1 -- 172.21.15.67:0/624507848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e4009400 con 0x7f12f00a8890 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.942+0000 7f12ee59c640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8890 0x7f12f01377d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c450 tx=0x7f12e000c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.942+0000 7f12deffd640 10 client.?.objecter ms_handle_connect 0x7f12f00a8890 2023-12-01T17:03:26.945 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.942+0000 7f12deffd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:26.954 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 1 -- 172.21.15.67:0/624507848 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e00052d0 con 0x7f12f00a8890 2023-12-01T17:03:26.954 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:26.954 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:26.954 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 10 monclient: dump: 2023-12-01T17:03:26.954 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:26.954 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 10 monclient: _finish_auth 0 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 10 monclient: _check_auth_tickets 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 1 -- 172.21.15.67:0/624507848 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0005430 con 0x7f12f00a8890 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12f4b8a640 5 monclient: authenticate success, global_id 4225 2023-12-01T17:03:26.955 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12f4b8a640 10 monclient: _renew_subs 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12f4b8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f0136a60 con 0x7f12f00a8890 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12f4b8a640 10 client.4225.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12ecd99640 4 set_mon_vals no callback set 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12deffd640 1 -- 172.21.15.67:0/624507848 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0012870 con 0x7f12f00a8890 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12f4b8a640 10 monclient: _renew_subs 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.952+0000 7f12f4b8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.953+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f0137d10 con 0x7f12f00a8890 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.953+0000 7f12f4b8a640 1 librados: init done 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.953+0000 7f12deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.953+0000 7f12deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.953+0000 7f12deffd640 10 monclient: dump: 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:26.956 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.954+0000 7f12f4b8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.954+0000 7f12f4b8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.954+0000 7f12f4b8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:26.957 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.954+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f0136ca0 con 0x7f12f00a8890 2023-12-01T17:03:26.987 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.987+0000 7f12deffd640 1 -- 172.21.15.67:0/624507848 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f12e00129d0 con 0x7f12f00a8890 2023-12-01T17:03:26.988 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.988+0000 7f12deffd640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f12d006c7e0 0x7f12d006eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:26.988 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.988+0000 7f12edd9b640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f12d006c7e0 0x7f12d006eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:26.988 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.988+0000 7f12edd9b640 10 monclient: get_auth_request con 0x7f12d006c7e0 auth_method 0 2023-12-01T17:03:26.989 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.988+0000 7f12deffd640 10 client.4225.objecter ms_handle_connect 0x7f12d006c7e0 2023-12-01T17:03:26.989 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:26.988+0000 7f12edd9b640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f12d006c7e0 0x7f12d006eca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12e4036000 tx=0x7f12e4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:27.044 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:17.428 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:27.137 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:17.539 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:27.421 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 1 -- 172.21.15.67:0/624507848 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f12e0058b60 con 0x7f12f00a8890 2023-12-01T17:03:27.421 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 10 client.4225.objecter ms_dispatch 0x7f12f0000b90 osd_map(10..10 src has 1..10) v4 2023-12-01T17:03:27.421 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 3 client.4225.objecter handle_osd_map got epochs [10,10] > 0 2023-12-01T17:03:27.421 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 3 client.4225.objecter handle_osd_map decoding full epoch 10 2023-12-01T17:03:27.422 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 20 client.4225.objecter dump_active .. 0 homeless 2023-12-01T17:03:27.422 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 1 -- 172.21.15.67:0/624507848 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12e00547a0 con 0x7f12f00a8890 2023-12-01T17:03:27.422 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:27.422 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.420+0000 7f12deffd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:27.474 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:17.876 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:27.623 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:18.016 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:27.642 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.642+0000 7f12f4b8a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:27.643 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.642+0000 7f12f4b8a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:27.643 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.642+0000 7f12f4b8a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:27.643 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.642+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12f0137ea0 con 0x7f12f00a8890 2023-12-01T17:03:27.668 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.668+0000 7f12deffd640 1 -- 172.21.15.67:0/624507848 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f12e0057dc0 con 0x7f12f00a8890 2023-12-01T17:03:27.668 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.668+0000 7f12deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:27.669 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.668+0000 7f12deffd640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:27.669 INFO:teuthology.misc.health.smithi067.stdout: 2023-12-01T17:03:27.669 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:09.418337+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:27.670 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 10 librados: watch_flush enter 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 10 librados: watch_flush exit 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 20 client.4225.objecter shutdown clearing up homeless session... 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 10 client.4225.objecter successfully canceled tick 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f12d006c7e0 msgr2=0x7f12d006eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f12d006c7e0 0x7f12d006eca0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f12e4036000 tx=0x7f12e4008030 comp rx=0 tx=0).stop 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 10 monclient: shutdown 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8890 msgr2=0x7f12f01377d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8890 0x7f12f01377d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f12e000c450 tx=0x7f12e000c910 comp rx=0 tx=0).stop 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 shutdown_connections 2023-12-01T17:03:27.671 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f12f013f0a0 0x7f12f0139b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:27.672 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f12d006c7e0 0x7f12d006eca0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:27.672 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f12f00a8c70 0x7f12f0139650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:27.672 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 --2- 172.21.15.67:0/624507848 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f12f00a8890 0x7f12f01377d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:27.672 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 >> 172.21.15.67:0/624507848 conn(0x7f12f0066db0 msgr2=0x7f12f0135db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:27.672 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 shutdown_connections 2023-12-01T17:03:27.672 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 -- 172.21.15.67:0/624507848 wait complete. 2023-12-01T17:03:27.672 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:27.670+0000 7f12f4b8a640 1 librados: shutdown 2023-12-01T17:03:27.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-01T17:03:28.159 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:18.562 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:28.229 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:18.609 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:28.846 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:19.234 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:28.959 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:19.347 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:32.128 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:22.512 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:32.132 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:22.517 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:32.632 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:23.035 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:32.636 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:23.038 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:32.636 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:23.038 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:32.640 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:23.042 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:33.000 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:23.392 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:33.004 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:23.397 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:33.752 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:24.132 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:33.756 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:24.136 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:33.829 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:24.232 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:33.833 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:24.237 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:34.405 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:24.792 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:34.409 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:24.797 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:34.440 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:24.827 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:34.444 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:24.832 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:34.487 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:34.593 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.592+0000 7f774972c640 10 monclient: get_monmap_and_config 2023-12-01T17:03:34.593 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.592+0000 7f774972c640 10 monclient: build_initial_monmap 2023-12-01T17:03:34.593 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.592+0000 7f774972c640 10 monclient: monmap: 2023-12-01T17:03:34.593 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:34.593 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:34.593 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:03:34.593426+0000 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:03:34.593426+0000 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:34.594 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:34.597 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.593+0000 7f774972c640 1 Processor -- start 2023-12-01T17:03:34.597 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.593+0000 7f774972c640 1 -- start start 2023-12-01T17:03:34.597 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.593+0000 7f774972c640 10 monclient: init 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f77440a8900 0x7f774413efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): picked mon.noname-c con 0x7f77440a8900 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a9b80 0x7f774413f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): picked mon.noname-b con 0x7f77440a9b80 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f77440a8ce0 0x7f77441453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): picked mon.noname-a con 0x7f77440a8ce0 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:34.599 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77440b1b80 con 0x7f77440a8ce0 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774413fa50 con 0x7f77440a8900 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77440aa870 con 0x7f77440a9b80 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.594+0000 7f774972c640 10 monclient(hunting): authenticate will time out at 1345.832020s 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.595+0000 7f773bfff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a9b80 0x7f774413f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.595+0000 7f773bfff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a9b80 0x7f774413f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:35882/0 (socket says 172.21.15.67:35882) 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.595+0000 7f773bfff640 1 -- 172.21.15.67:0/548675845 learned_addr learned my addr 172.21.15.67:0/548675845 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.595+0000 7f773bfff640 10 monclient(hunting): get_auth_request con 0x7f77440a9b80 auth_method 0 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.595+0000 7f773bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.595+0000 7f773bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.595+0000 7f773bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:34.600 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.596+0000 7f773bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.596+0000 7f773bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.596+0000 7f773bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.597+0000 7f773bfff640 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.597+0000 7f773bfff640 1 -- 172.21.15.67:0/548675845 >> v2:172.21.15.92:3300/0 conn(0x7f77440a8900 msgr2=0x7f774413efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.597+0000 7f773bfff640 1 --2- 172.21.15.67:0/548675845 >> v2:172.21.15.92:3300/0 conn(0x7f77440a8900 0x7f774413efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.597+0000 7f773bfff640 1 -- 172.21.15.67:0/548675845 >> v2:172.21.15.67:3300/0 conn(0x7f77440a8ce0 msgr2=0x7f77441453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.597+0000 7f773bfff640 1 --2- 172.21.15.67:0/548675845 >> v2:172.21.15.67:3300/0 conn(0x7f77440a8ce0 0x7f77441453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.597+0000 7f773bfff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.597+0000 7f773bfff640 1 monclient: found mon.noname-b 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.598+0000 7f773bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:34.601 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.598+0000 7f773bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:03:34.602 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.598+0000 7f773bfff640 1 -- 172.21.15.67:0/548675845 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77440aab50 con 0x7f77440a9b80 2023-12-01T17:03:34.605 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.600+0000 7f773bfff640 1 --2- 172.21.15.67:0/548675845 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a9b80 0x7f774413f510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f77440b4fb0 tx=0x7f772c02b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d1714bc3848d37b server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:34.605 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f77427fc640 1 -- 172.21.15.67:0/548675845 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c031860 con 0x7f77440a9b80 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f77427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f77427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f77427fc640 10 monclient: dump: 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:34.606 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f77427fc640 10 monclient: _finish_auth 0 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f77427fc640 10 monclient: _check_auth_tickets 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f77427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f774972c640 5 monclient: authenticate success, global_id 4181 2023-12-01T17:03:34.607 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.605+0000 7f774972c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:34.608 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f77427fc640 1 -- 172.21.15.67:0/548675845 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c0319c0 con 0x7f77440a9b80 2023-12-01T17:03:34.608 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f77427fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f774972c640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f774972c640 4 set_mon_vals no callback set 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f77427fc640 1 -- 172.21.15.67:0/548675845 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c031d50 con 0x7f77440a9b80 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f77427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f77427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.607+0000 7f77427fc640 10 monclient: dump: 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:34.609 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 10 monclient: shutdown 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 -- 172.21.15.67:0/548675845 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a9b80 msgr2=0x7f774413f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 --2- 172.21.15.67:0/548675845 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a9b80 0x7f774413f510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f77440b4fb0 tx=0x7f772c02b900 comp rx=0 tx=0).stop 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 -- 172.21.15.67:0/548675845 shutdown_connections 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 --2- 172.21.15.67:0/548675845 >> v2:172.21.15.67:3300/0 conn(0x7f77440a8ce0 0x7f77441453f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 --2- 172.21.15.67:0/548675845 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a9b80 0x7f774413f510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 --2- 172.21.15.67:0/548675845 >> v2:172.21.15.92:3300/0 conn(0x7f77440a8900 0x7f774413efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 -- 172.21.15.67:0/548675845 >> 172.21.15.67:0/548675845 conn(0x7f7744066f00 msgr2=0x7f77440b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 -- 172.21.15.67:0/548675845 shutdown_connections 2023-12-01T17:03:34.610 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.608+0000 7f774972c640 1 -- 172.21.15.67:0/548675845 wait complete. 2023-12-01T17:03:34.615 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.609+0000 7f774972c640 10 monclient: build_initial_monmap 2023-12-01T17:03:34.615 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.609+0000 7f774972c640 10 monclient: monmap: 2023-12-01T17:03:34.615 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:34.615 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:34.615 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:34.615 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.609+0000 7f774972c640 1 librados: starting msgr at 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.610+0000 7f774972c640 1 librados: starting objecter 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.610+0000 7f774972c640 1 Processor -- start 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.610+0000 7f774972c640 1 -- start start 2023-12-01T17:03:34.616 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.610+0000 7f774972c640 1 librados: setting wanted keys 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.610+0000 7f774972c640 1 librados: calling monclient init 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.610+0000 7f774972c640 10 monclient: init 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77440a8900 0x7f77440abab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 10 monclient(hunting): picked mon.noname-b con 0x7f77440a8900 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a8ce0 0x7f77440ac830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 10 monclient(hunting): picked mon.noname-c con 0x7f77440a8ce0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440a9b80 0x7f7744138990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:34.617 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.611+0000 7f774972c640 10 monclient(hunting): picked mon.noname-a con 0x7f77440a9b80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77440ad970 con 0x7f77440a9b80 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77441d71b0 con 0x7f77440a8900 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7744142270 con 0x7f77440a8ce0 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f774972c640 10 monclient(hunting): authenticate will time out at 1345.849834s 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f773bfff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a8ce0 0x7f77440ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f773bfff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a8ce0 0x7f77440ac830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:35894/0 (socket says 172.21.15.67:35894) 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f773bfff640 1 -- 172.21.15.67:0/2331746106 learned_addr learned my addr 172.21.15.67:0/2331746106 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.612+0000 7f773bfff640 10 monclient(hunting): get_auth_request con 0x7f77440a8ce0 auth_method 0 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.613+0000 7f773bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:34.618 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.613+0000 7f773bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.613+0000 7f773bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f773bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f773bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f773bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f7742ffd640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77440a8900 0x7f77440abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f7742ffd640 10 monclient(hunting): get_auth_request con 0x7f77440a8900 auth_method 0 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f7742ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f7742ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:34.619 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.614+0000 7f7742ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.616+0000 7f7742ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.616+0000 7f7742ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.616+0000 7f7742ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 10 monclient(hunting): handle_auth_done global_id 4184 payload 824 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 1 -- 172.21.15.67:0/2331746106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77440a8900 msgr2=0x7f77440abab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77440a8900 0x7f77440abab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 1 -- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440a9b80 msgr2=0x7f7744138990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:34.620 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440a9b80 0x7f7744138990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:34.621 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:34.621 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 1 monclient: found mon.noname-c 2023-12-01T17:03:34.621 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:34.621 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:03:34.621 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.617+0000 7f773bfff640 1 -- 172.21.15.67:0/2331746106 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f772c0094d0 con 0x7f77440a8ce0 2023-12-01T17:03:34.625 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.625+0000 7f773bfff640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a8ce0 0x7f77440ac830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f772c000c00 tx=0x7f772c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:34.626 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.625+0000 7f773b7fe640 10 client.?.objecter ms_handle_connect 0x7f77440a8ce0 2023-12-01T17:03:34.626 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.625+0000 7f773b7fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:34.673 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 1 -- 172.21.15.67:0/2331746106 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c03ad10 con 0x7f77440a8ce0 2023-12-01T17:03:34.674 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:34.674 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:34.674 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: dump: 2023-12-01T17:03:34.674 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:34.674 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:34.675 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.675 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.675 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:34.675 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:34.675 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:34.675 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:34.675 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:34.676 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:34.676 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: _finish_auth 0 2023-12-01T17:03:34.676 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:03:34.676 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:34.676 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 1 -- 172.21.15.67:0/2331746106 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c03ae70 con 0x7f77440a8ce0 2023-12-01T17:03:34.676 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:34.677 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 1 -- 172.21.15.67:0/2331746106 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c044610 con 0x7f77440a8ce0 2023-12-01T17:03:34.677 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:34.677 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:34.677 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f773b7fe640 10 monclient: dump: 2023-12-01T17:03:34.677 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:34.677 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f7741ffb640 4 set_mon_vals no callback set 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f774972c640 5 monclient: authenticate success, global_id 4184 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f774972c640 10 monclient: _renew_subs 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f774972c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:34.678 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.672+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77440adbc0 con 0x7f77440a8ce0 2023-12-01T17:03:34.679 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.673+0000 7f774972c640 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:34.679 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.673+0000 7f774972c640 10 monclient: _renew_subs 2023-12-01T17:03:34.679 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.673+0000 7f774972c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:34.679 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.673+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7744138ed0 con 0x7f77440a8ce0 2023-12-01T17:03:34.679 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.673+0000 7f774972c640 1 librados: init done 2023-12-01T17:03:34.680 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.674+0000 7f774972c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:34.680 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.674+0000 7f774972c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:34.680 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.674+0000 7f774972c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:34.680 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.674+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77441d7340 con 0x7f77440a8ce0 2023-12-01T17:03:34.696 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.694+0000 7f773b7fe640 1 -- 172.21.15.67:0/2331746106 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f772c04d450 con 0x7f77440a8ce0 2023-12-01T17:03:34.696 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.695+0000 7f773b7fe640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f772406c7c0 0x7f772406ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:34.696 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.695+0000 7f7742ffd640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f772406c7c0 0x7f772406ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:34.696 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.695+0000 7f7742ffd640 10 monclient: get_auth_request con 0x7f772406c7c0 auth_method 0 2023-12-01T17:03:34.696 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.696+0000 7f773b7fe640 10 client.4184.objecter ms_handle_connect 0x7f772406c7c0 2023-12-01T17:03:34.696 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.696+0000 7f7742ffd640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f772406c7c0 0x7f772406ec80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7734009990 tx=0x7f7734009110 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:34.701 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.698+0000 7f773b7fe640 1 -- 172.21.15.67:0/2331746106 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f772c07b400 con 0x7f77440a8ce0 2023-12-01T17:03:34.701 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.698+0000 7f773b7fe640 10 client.4184.objecter ms_dispatch 0x7f7744000b90 osd_map(10..10 src has 1..10) v4 2023-12-01T17:03:34.701 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.698+0000 7f773b7fe640 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2023-12-01T17:03:34.701 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.698+0000 7f773b7fe640 3 client.4184.objecter handle_osd_map decoding full epoch 10 2023-12-01T17:03:34.702 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:34.698+0000 7f773b7fe640 20 client.4184.objecter dump_active .. 0 homeless 2023-12-01T17:03:35.017 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.017+0000 7f773b7fe640 1 -- 172.21.15.67:0/2331746106 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f772c077040 con 0x7f77440a8ce0 2023-12-01T17:03:35.017 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.017+0000 7f773b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:35.017 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.017+0000 7f773b7fe640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:35.245 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.244+0000 7f774972c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:35.245 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.244+0000 7f774972c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:35.245 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.244+0000 7f774972c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:35.247 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.244+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f774406b320 con 0x7f77440a8ce0 2023-12-01T17:03:35.273 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.273+0000 7f773b7fe640 1 -- 172.21.15.67:0/2331746106 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f772c07a660 con 0x7f77440a8ce0 2023-12-01T17:03:35.274 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.273+0000 7f773b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:35.274 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.273+0000 7f773b7fe640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:35.274 INFO:teuthology.misc.health.smithi067.stdout: 2023-12-01T17:03:35.274 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:09.418337+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:35.288 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 10 librados: watch_flush enter 2023-12-01T17:03:35.288 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 10 librados: watch_flush exit 2023-12-01T17:03:35.288 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 20 client.4184.objecter shutdown clearing up homeless session... 2023-12-01T17:03:35.288 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 10 client.4184.objecter successfully canceled tick 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f772406c7c0 msgr2=0x7f772406ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f772406c7c0 0x7f772406ec80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7734009990 tx=0x7f7734009110 comp rx=0 tx=0).stop 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 10 monclient: shutdown 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a8ce0 msgr2=0x7f77440ac830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a8ce0 0x7f77440ac830 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f772c000c00 tx=0x7f772c0046c0 comp rx=0 tx=0).stop 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 shutdown_connections 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77440a9b80 0x7f7744138990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f772406c7c0 0x7f772406ec80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77440a8ce0 0x7f77440ac830 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 --2- 172.21.15.67:0/2331746106 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77440a8900 0x7f77440abab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.275+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 >> 172.21.15.67:0/2331746106 conn(0x7f7744066f00 msgr2=0x7f7744139820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.276+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 shutdown_connections 2023-12-01T17:03:35.289 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.276+0000 7f774972c640 1 -- 172.21.15.67:0/2331746106 wait complete. 2023-12-01T17:03:35.290 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:35.276+0000 7f774972c640 1 librados: shutdown 2023-12-01T17:03:35.291 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-01T17:03:35.968 INFO:tasks.ceph.osd.7.smithi092.stderr:==00:00:00:26.352 87395== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:36.614 INFO:tasks.ceph.osd.6.smithi092.stderr:==00:00:00:27.016 87393== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:36.618 INFO:tasks.ceph.osd.5.smithi092.stderr:==00:00:00:27.020 87392== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:37.235 INFO:tasks.ceph.osd.4.smithi092.stderr:==00:00:00:27.627 87394== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:37.935 INFO:tasks.ceph.osd.0.smithi067.stderr:==00:00:00:28.339 86890== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:38.380 INFO:tasks.ceph.osd.3.smithi067.stderr:==00:00:00:28.760 86893== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:38.926 INFO:tasks.ceph.osd.2.smithi067.stderr:==00:00:00:29.314 86891== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:39.136 INFO:tasks.ceph.osd.1.smithi067.stderr:==00:00:00:29.524 86892== Warning: unimplemented fcntl command: 1038 2023-12-01T17:03:42.097 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:42.212 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.212+0000 7f099cb97640 10 monclient: get_monmap_and_config 2023-12-01T17:03:42.212 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.212+0000 7f099cb97640 10 monclient: build_initial_monmap 2023-12-01T17:03:42.212 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.212+0000 7f099cb97640 10 monclient: monmap: 2023-12-01T17:03:42.212 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:42.212 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:03:42.212860+0000 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:03:42.212860+0000 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:42.213 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:42.214 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.212+0000 7f099cb97640 1 Processor -- start 2023-12-01T17:03:42.214 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.213+0000 7f099cb97640 1 -- start start 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.213+0000 7f099cb97640 10 monclient: init 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09980a8900 0x7f099813efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient(hunting): picked mon.noname-c con 0x7f09980a8900 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09980a9b80 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f09981453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient(hunting): picked mon.noname-b con 0x7f09980a8ce0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09980b1b80 con 0x7f09980a9b80 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09980aa870 con 0x7f09980a8900 2023-12-01T17:03:42.215 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:42.217 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09980aab50 con 0x7f09980a8ce0 2023-12-01T17:03:42.217 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:42.217 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.214+0000 7f099cb97640 10 monclient(hunting): authenticate will time out at 1353.451716s 2023-12-01T17:03:42.219 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f099659c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09980a8900 0x7f099813efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f0996d9d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f09981453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f099659c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f09980a8900 0x7f099813efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:48310/0 (socket says 172.21.15.67:48310) 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f099659c640 1 -- 172.21.15.67:0/1740365439 learned_addr learned my addr 172.21.15.67:0/1740365439 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f099659c640 10 monclient(hunting): get_auth_request con 0x7f09980a8900 auth_method 0 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f099659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f099659c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.215+0000 7f099659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f0996d9d640 10 monclient(hunting): get_auth_request con 0x7f09980a8ce0 auth_method 0 2023-12-01T17:03:42.220 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f0996d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f0996d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f0996d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f099659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f099659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f099659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f0996d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f0996d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.216+0000 7f0996d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 10 monclient(hunting): handle_auth_done global_id 4187 payload 274 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 1 -- 172.21.15.67:0/1740365439 >> v2:172.21.15.92:3300/0 conn(0x7f09980a8900 msgr2=0x7f099813efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:42.221 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 1 --2- 172.21.15.67:0/1740365439 >> v2:172.21.15.92:3300/0 conn(0x7f09980a8900 0x7f099813efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.222 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 1 -- 172.21.15.67:0/1740365439 >> v1:172.21.15.92:6789/0 conn(0x7f09980a9b80 legacy=0x7f099813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:42.222 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:03:42.222 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 1 monclient: found mon.noname-b 2023-12-01T17:03:42.222 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:42.222 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:03:42.222 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.219+0000 7f0996d9d640 1 -- 172.21.15.67:0/1740365439 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0998145930 con 0x7f09980a8ce0 2023-12-01T17:03:42.223 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.221+0000 7f0996d9d640 1 --2- 172.21.15.67:0/1740365439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f09981453f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f099002be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf6c23d22b99e3e4 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 1 -- 172.21.15.67:0/1740365439 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099002cbc0 con 0x7f09980a8ce0 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: dump: 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:42.227 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: _finish_auth 0 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: _check_auth_tickets 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 1 -- 172.21.15.67:0/1740365439 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099002cd20 con 0x7f09980a8ce0 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099cb97640 5 monclient: authenticate success, global_id 4187 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099cb97640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099cb97640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 1 -- 172.21.15.67:0/1740365439 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990034660 con 0x7f09980a8ce0 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099cb97640 4 set_mon_vals no callback set 2023-12-01T17:03:42.228 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.227+0000 7f099559a640 10 monclient: dump: 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:42.229 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 10 monclient: shutdown 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 -- 172.21.15.67:0/1740365439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 msgr2=0x7f09981453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 --2- 172.21.15.67:0/1740365439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f09981453f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0990009d90 tx=0x7f099002be10 comp rx=0 tx=0).stop 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 -- 172.21.15.67:0/1740365439 shutdown_connections 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 --2- 172.21.15.67:0/1740365439 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f09981453f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 --2- 172.21.15.67:0/1740365439 >> v2:172.21.15.92:3300/0 conn(0x7f09980a8900 0x7f099813efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.234 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 -- 172.21.15.67:0/1740365439 >> 172.21.15.67:0/1740365439 conn(0x7f0998066f00 msgr2=0x7f09980b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 -- 172.21.15.67:0/1740365439 shutdown_connections 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.228+0000 7f099cb97640 1 -- 172.21.15.67:0/1740365439 wait complete. 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.229+0000 7f099cb97640 10 monclient: build_initial_monmap 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.229+0000 7f099cb97640 10 monclient: monmap: 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:42.235 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.229+0000 7f099cb97640 1 librados: starting msgr at 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.229+0000 7f099cb97640 1 librados: starting objecter 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.230+0000 7f099cb97640 1 Processor -- start 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.230+0000 7f099cb97640 1 -- start start 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.230+0000 7f099cb97640 1 librados: setting wanted keys 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.230+0000 7f099cb97640 1 librados: calling monclient init 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.230+0000 7f099cb97640 10 monclient: init 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.230+0000 7f099cb97640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.230+0000 7f099cb97640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09980a8900 0x7f09981347c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): picked mon.noname-a con 0x7f09980a8900 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:42.236 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f0998144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): picked mon.noname-c con 0x7f09980a8ce0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09980a9b80 0x7f0998137070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): picked mon.noname-b con 0x7f09980a9b80 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0998144300 con 0x7f09980a8900 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09981445e0 con 0x7f09980a9b80 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0998138240 con 0x7f09980a8ce0 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099cb97640 10 monclient(hunting): authenticate will time out at 1353.468704s 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f099659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09980a8900 0x7f09981347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f0995d9b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f0998144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.231+0000 7f0996d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09980a9b80 0x7f0998137070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:42.237 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09980a8900 0x7f09981347c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51724/0 (socket says 172.21.15.67:51724) 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 1 -- 172.21.15.67:0/2216500664 learned_addr learned my addr 172.21.15.67:0/2216500664 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 10 monclient(hunting): get_auth_request con 0x7f09980a8900 auth_method 0 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0995d9b640 10 monclient(hunting): get_auth_request con 0x7f09980a8ce0 auth_method 0 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0995d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0995d9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0995d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0996d9d640 10 monclient(hunting): get_auth_request con 0x7f09980a9b80 auth_method 0 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0996d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0996d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f0996d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:42.238 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.232+0000 7f099659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:42.239 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.233+0000 7f099659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:42.239 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.233+0000 7f0995d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:42.239 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.233+0000 7f0995d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:42.239 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.233+0000 7f0995d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:42.239 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.233+0000 7f0996d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:42.239 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.233+0000 7f0996d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:42.239 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.233+0000 7f0996d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:42.240 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.236+0000 7f0995d9b640 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2023-12-01T17:03:42.240 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.236+0000 7f0995d9b640 1 -- 172.21.15.67:0/2216500664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09980a9b80 msgr2=0x7f0998137070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.236+0000 7f0995d9b640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09980a9b80 0x7f0998137070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.236+0000 7f0995d9b640 1 -- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09980a8900 msgr2=0x7f09981347c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.237+0000 7f0995d9b640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09980a8900 0x7f09981347c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.237+0000 7f0995d9b640 10 monclient: _finish_hunting 0 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.237+0000 7f0995d9b640 1 monclient: found mon.noname-c 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.237+0000 7f0995d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.237+0000 7f0995d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.237+0000 7f0995d9b640 1 -- 172.21.15.67:0/2216500664 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990009a70 con 0x7f09980a8ce0 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.240+0000 7f0995d9b640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f0998144e10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f098c00b3e0 tx=0x7f098c00b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.240+0000 7f0986ffd640 10 client.?.objecter ms_handle_connect 0x7f09980a8ce0 2023-12-01T17:03:42.241 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.240+0000 7f0986ffd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.245+0000 7f0986ffd640 1 -- 172.21.15.67:0/2216500664 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c004020 con 0x7f09980a8ce0 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.245+0000 7f0986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: dump: 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.248 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: _finish_auth 0 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 1 -- 172.21.15.67:0/2216500664 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f098c004180 con 0x7f09980a8ce0 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 5 monclient: authenticate success, global_id 4190 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 10 monclient: _renew_subs 2023-12-01T17:03:42.249 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0998138580 con 0x7f09980a8ce0 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 1 -- 172.21.15.67:0/2216500664 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c011630 con 0x7f09980a8ce0 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0986ffd640 10 monclient: dump: 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:42.250 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f0994d99640 4 set_mon_vals no callback set 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 10 monclient: _renew_subs 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0998138c60 con 0x7f09980a8ce0 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.246+0000 7f099cb97640 1 librados: init done 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.247+0000 7f099cb97640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.247+0000 7f099cb97640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.247+0000 7f099cb97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:42.251 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.247+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f099806b320 con 0x7f09980a8ce0 2023-12-01T17:03:42.556 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.555+0000 7f0986ffd640 1 -- 172.21.15.67:0/2216500664 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f098c011790 con 0x7f09980a8ce0 2023-12-01T17:03:42.556 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.556+0000 7f0986ffd640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f097806c790 0x7f097806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:42.556 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.556+0000 7f099659c640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f097806c790 0x7f097806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:42.556 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.556+0000 7f099659c640 10 monclient: get_auth_request con 0x7f097806c790 auth_method 0 2023-12-01T17:03:42.557 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.556+0000 7f0986ffd640 10 client.4190.objecter ms_handle_connect 0x7f097806c790 2023-12-01T17:03:42.557 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.556+0000 7f099659c640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f097806c790 0x7f097806ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f09880094a0 tx=0x7f0988009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:42.580 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 1 -- 172.21.15.67:0/2216500664 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f098c058740 con 0x7f09980a8ce0 2023-12-01T17:03:42.580 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 10 client.4190.objecter ms_dispatch 0x7f0998000b90 osd_map(10..10 src has 1..10) v4 2023-12-01T17:03:42.580 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2023-12-01T17:03:42.580 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2023-12-01T17:03:42.580 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 20 client.4190.objecter dump_active .. 0 homeless 2023-12-01T17:03:42.581 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 1 -- 172.21.15.67:0/2216500664 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f098c054a80 con 0x7f09980a8ce0 2023-12-01T17:03:42.581 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:42.581 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.580+0000 7f0986ffd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:42.916 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.915+0000 7f099cb97640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:42.916 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.915+0000 7f099cb97640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:42.916 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.915+0000 7f099cb97640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:42.916 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.915+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0998144720 con 0x7f09980a8ce0 2023-12-01T17:03:42.932 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.932+0000 7f0986ffd640 1 -- 172.21.15.67:0/2216500664 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0998144720 con 0x7f09980a8ce0 2023-12-01T17:03:42.932 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.932+0000 7f0986ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:42.932 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.932+0000 7f0986ffd640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:42.933 INFO:teuthology.misc.health.smithi067.stdout: 2023-12-01T17:03:42.933 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":10,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:09.418337+0000","last_up_change":"0.000000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:42.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.934+0000 7f099cb97640 10 librados: watch_flush enter 2023-12-01T17:03:42.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 10 librados: watch_flush exit 2023-12-01T17:03:42.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 20 client.4190.objecter shutdown clearing up homeless session... 2023-12-01T17:03:42.935 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 10 client.4190.objecter successfully canceled tick 2023-12-01T17:03:42.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f097806c790 msgr2=0x7f097806ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:42.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f097806c790 0x7f097806ec50 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f09880094a0 tx=0x7f0988009240 comp rx=0 tx=0).stop 2023-12-01T17:03:42.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 10 monclient: shutdown 2023-12-01T17:03:42.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:42.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 msgr2=0x7f0998144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:42.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f0998144e10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f098c00b3e0 tx=0x7f098c00b8a0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.937 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 shutdown_connections 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f09980a9b80 0x7f0998137070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f097806c790 0x7f097806ec50 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f09980a8ce0 0x7f0998144e10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 --2- 172.21.15.67:0/2216500664 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f09980a8900 0x7f09981347c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.935+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 >> 172.21.15.67:0/2216500664 conn(0x7f0998066f00 msgr2=0x7f099813d690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.936+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 shutdown_connections 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.936+0000 7f099cb97640 1 -- 172.21.15.67:0/2216500664 wait complete. 2023-12-01T17:03:42.938 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:42.936+0000 7f099cb97640 1 librados: shutdown 2023-12-01T17:03:42.947 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-12-01T17:03:44.246 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-12-01T17:03:44.243+0000 5dc2340 -1 osd.7 0 log_to_monitors true 2023-12-01T17:03:44.873 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:03:44.870+0000 5dc2340 -1 osd.5 0 log_to_monitors true 2023-12-01T17:03:45.060 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:03:45.058+0000 5dc2340 -1 osd.6 0 log_to_monitors true 2023-12-01T17:03:45.277 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:03:45.274+0000 5dc2340 -1 osd.4 0 log_to_monitors true 2023-12-01T17:03:46.444 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:03:46.441+0000 5dc2340 -1 osd.0 0 log_to_monitors true 2023-12-01T17:03:46.772 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:03:46.769+0000 5dc2340 -1 osd.3 0 log_to_monitors true 2023-12-01T17:03:47.452 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-12-01T17:03:47.449+0000 c493640 -1 osd.7 0 waiting for initial osdmap 2023-12-01T17:03:47.522 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:03:47.517+0000 5dc2340 -1 osd.2 0 log_to_monitors true 2023-12-01T17:03:47.799 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-12-01T17:03:47.786+0000 12a57640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:47.978 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:03:47.975+0000 5dc2340 -1 osd.1 0 log_to_monitors true 2023-12-01T17:03:48.743 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:03:48.731+0000 ac7d640 -1 osd.6 0 waiting for initial osdmap 2023-12-01T17:03:48.756 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:03:48.754+0000 ac7d640 -1 osd.5 0 waiting for initial osdmap 2023-12-01T17:03:48.764 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:03:48.760+0000 c493640 -1 osd.4 0 waiting for initial osdmap 2023-12-01T17:03:49.110 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:03:49.096+0000 13a59640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:49.110 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:03:49.095+0000 12256640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:49.190 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:03:49.171+0000 13258640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:49.717 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:03:49.703+0000 c493640 -1 osd.0 0 waiting for initial osdmap 2023-12-01T17:03:49.753 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:49.865 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.864+0000 7fbf59e56640 10 monclient: get_monmap_and_config 2023-12-01T17:03:49.865 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.864+0000 7fbf59e56640 10 monclient: build_initial_monmap 2023-12-01T17:03:49.865 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.864+0000 7fbf59e56640 10 monclient: monmap: 2023-12-01T17:03:49.865 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:49.865 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:49.865 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:03:49.865645+0000 2023-12-01T17:03:49.865 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:03:49.865645+0000 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:49.866 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.865+0000 7fbf59e56640 1 Processor -- start 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.865+0000 7fbf59e56640 1 -- start start 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.865+0000 7fbf59e56640 10 monclient: init 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8900 0x7fbf5413efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): picked mon.noname-b con 0x7fbf540a8900 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf540a9b80 0x7fbf5413f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): picked mon.noname-a con 0x7fbf540a9b80 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbf540a8ce0 0x7fbf541453f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:49.869 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): picked mon.noname-c con 0x7fbf540a8ce0 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf540b1b80 con 0x7fbf540a9b80 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5413fa50 con 0x7fbf540a8ce0 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540aa870 con 0x7fbf540a8900 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.866+0000 7fbf59e56640 10 monclient(hunting): authenticate will time out at 1361.103814s 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbf540a8ce0 0x7fbf541453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf537fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8900 0x7fbf5413efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf52ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbf540a9b80 0x7fbf5413f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbf540a8ce0 0x7fbf541453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:48384/0 (socket says 172.21.15.67:48384) 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 1 -- 172.21.15.67:0/1137790367 learned_addr learned my addr 172.21.15.67:0/1137790367 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 10 monclient(hunting): get_auth_request con 0x7fbf540a8ce0 auth_method 0 2023-12-01T17:03:49.870 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf537fe640 10 monclient(hunting): get_auth_request con 0x7fbf540a8900 auth_method 0 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf537fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf52ffd640 10 monclient(hunting): get_auth_request con 0x7fbf540a9b80 auth_method 0 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.867+0000 7fbf53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:49.871 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.868+0000 7fbf537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:49.872 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.868+0000 7fbf537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:49.872 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.868+0000 7fbf537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:49.872 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.868+0000 7fbf52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:49.872 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.868+0000 7fbf52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:49.872 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.868+0000 7fbf52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:49.872 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 10 monclient(hunting): handle_auth_done global_id 4255 payload 274 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 1 -- 172.21.15.67:0/1137790367 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8900 msgr2=0x7fbf5413efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 1 --2- 172.21.15.67:0/1137790367 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8900 0x7fbf5413efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 1 -- 172.21.15.67:0/1137790367 >> v2:172.21.15.67:3300/0 conn(0x7fbf540a9b80 msgr2=0x7fbf5413f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 1 --2- 172.21.15.67:0/1137790367 >> v2:172.21.15.67:3300/0 conn(0x7fbf540a9b80 0x7fbf5413f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 1 monclient: found mon.noname-c 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.869+0000 7fbf53fff640 1 -- 172.21.15.67:0/1137790367 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf540aab50 con 0x7fbf540a8ce0 2023-12-01T17:03:49.873 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.872+0000 7fbf53fff640 1 --2- 172.21.15.67:0/1137790367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8ce0 0x7fbf541453f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbf540b50a0 tx=0x7fbf4c02bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d45ceed77118bb1d server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:49.876 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 1 -- 172.21.15.67:0/1137790367 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c02ca70 con 0x7fbf540a8ce0 2023-12-01T17:03:49.876 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:49.876 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:49.876 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: dump: 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: _finish_auth 0 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: _check_auth_tickets 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:49.877 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 1 -- 172.21.15.67:0/1137790367 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c02cbd0 con 0x7fbf540a8ce0 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf59e56640 5 monclient: authenticate success, global_id 4255 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf59e56640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf59e56640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf59e56640 4 set_mon_vals no callback set 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 1 -- 172.21.15.67:0/1137790367 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c0343f0 con 0x7fbf540a8ce0 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.876+0000 7fbf527fc640 10 monclient: dump: 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.878 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:49.879 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:49.879 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:49.879 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:49.879 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:49.879 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 10 monclient: shutdown 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 -- 172.21.15.67:0/1137790367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8ce0 msgr2=0x7fbf541453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 --2- 172.21.15.67:0/1137790367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8ce0 0x7fbf541453f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbf540b50a0 tx=0x7fbf4c02bed0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 -- 172.21.15.67:0/1137790367 shutdown_connections 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 --2- 172.21.15.67:0/1137790367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8ce0 0x7fbf541453f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 --2- 172.21.15.67:0/1137790367 >> v2:172.21.15.67:3300/0 conn(0x7fbf540a9b80 0x7fbf5413f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.882 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 --2- 172.21.15.67:0/1137790367 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8900 0x7fbf5413efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 -- 172.21.15.67:0/1137790367 >> 172.21.15.67:0/1137790367 conn(0x7fbf54066f00 msgr2=0x7fbf540b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 -- 172.21.15.67:0/1137790367 shutdown_connections 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 1 -- 172.21.15.67:0/1137790367 wait complete. 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 10 monclient: build_initial_monmap 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.877+0000 7fbf59e56640 10 monclient: monmap: 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:49.883 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.878+0000 7fbf59e56640 1 librados: starting msgr at 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.878+0000 7fbf59e56640 1 librados: starting objecter 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.878+0000 7fbf59e56640 1 Processor -- start 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.878+0000 7fbf59e56640 1 -- start start 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.878+0000 7fbf59e56640 1 librados: setting wanted keys 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.878+0000 7fbf59e56640 1 librados: calling monclient init 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.878+0000 7fbf59e56640 10 monclient: init 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8900 0x7fbf540ab580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): picked mon.noname-b con 0x7fbf540a8900 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:49.884 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8ce0 0x7fbf541d77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): picked mon.noname-c con 0x7fbf540a8ce0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf540a9b80 0x7fbf541da640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): picked mon.noname-a con 0x7fbf540a9b80 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf541d70c0 con 0x7fbf540a9b80 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf540acb70 con 0x7fbf540a8900 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf541d72d0 con 0x7fbf540a8ce0 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf59e56640 10 monclient(hunting): authenticate will time out at 1361.116598s 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf537fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8900 0x7fbf540ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.879+0000 7fbf52ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8ce0 0x7fbf541d77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8900 0x7fbf540ab580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:48388/0 (socket says 172.21.15.67:48388) 2023-12-01T17:03:49.885 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 1 -- 172.21.15.67:0/625748571 learned_addr learned my addr 172.21.15.67:0/625748571 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 10 monclient(hunting): get_auth_request con 0x7fbf540a8900 auth_method 0 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf52ffd640 10 monclient(hunting): get_auth_request con 0x7fbf540a8ce0 auth_method 0 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf52ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf52ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.880+0000 7fbf537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:49.886 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.881+0000 7fbf52ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:49.887 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.881+0000 7fbf52ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:49.887 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.881+0000 7fbf52ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:49.887 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.882+0000 7fbf53fff640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf540a9b80 0x7fbf541da640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:49.888 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.882+0000 7fbf53fff640 10 monclient(hunting): get_auth_request con 0x7fbf540a9b80 auth_method 0 2023-12-01T17:03:49.888 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.882+0000 7fbf53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:49.888 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.882+0000 7fbf53fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:49.888 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.882+0000 7fbf53fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:49.888 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.883+0000 7fbf53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:49.888 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.883+0000 7fbf53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:49.888 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.883+0000 7fbf53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 10 monclient(hunting): handle_auth_done global_id 4214 payload 824 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 1 -- 172.21.15.67:0/625748571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8900 msgr2=0x7fbf540ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8900 0x7fbf540ab580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 1 -- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf540a9b80 msgr2=0x7fbf541da640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf540a9b80 0x7fbf541da640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 10 monclient: _finish_hunting 0 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 1 monclient: found mon.noname-c 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.884+0000 7fbf52ffd640 1 -- 172.21.15.67:0/625748571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf4c009b30 con 0x7fbf540a8ce0 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.887+0000 7fbf52ffd640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8ce0 0x7fbf541d77d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800e840 tx=0x7fbf4800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.887+0000 7fbf3bfff640 10 client.?.objecter ms_handle_connect 0x7fbf540a8ce0 2023-12-01T17:03:49.889 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.887+0000 7fbf3bfff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:49.889 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:03:49.882+0000 c493640 -1 osd.3 0 waiting for initial osdmap 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 1 -- 172.21.15.67:0/625748571 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800c730 con 0x7fbf540a8ce0 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: dump: 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:49.895 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: _finish_auth 0 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: _check_auth_tickets 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 1 -- 172.21.15.67:0/625748571 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4800c890 con 0x7fbf540a8ce0 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 1 -- 172.21.15.67:0/625748571 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4800cc40 con 0x7fbf540a8ce0 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 5 monclient: authenticate success, global_id 4214 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 10 monclient: _renew_subs 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:49.896 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf540abac0 con 0x7fbf540a8ce0 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf3bfff640 10 monclient: dump: 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:49.897 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:49.898 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:49.898 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf51ffb640 4 set_mon_vals no callback set 2023-12-01T17:03:49.898 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 10 client.4214.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:49.898 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 10 monclient: _renew_subs 2023-12-01T17:03:49.898 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:49.898 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf54144d90 con 0x7fbf540a8ce0 2023-12-01T17:03:49.898 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.894+0000 7fbf59e56640 1 librados: init done 2023-12-01T17:03:49.899 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.895+0000 7fbf59e56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:49.899 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.895+0000 7fbf59e56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:49.899 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.895+0000 7fbf59e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:49.899 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.895+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf541d7490 con 0x7fbf540a8ce0 2023-12-01T17:03:49.927 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.927+0000 7fbf3bfff640 1 -- 172.21.15.67:0/625748571 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fbf48004020 con 0x7fbf540a8ce0 2023-12-01T17:03:49.928 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.927+0000 7fbf3bfff640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbf4006c790 0x7fbf4006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:49.928 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.927+0000 7fbf537fe640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbf4006c790 0x7fbf4006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:49.928 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.928+0000 7fbf537fe640 10 monclient: get_auth_request con 0x7fbf4006c790 auth_method 0 2023-12-01T17:03:49.928 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.928+0000 7fbf3bfff640 10 client.4214.objecter ms_handle_connect 0x7fbf4006c790 2023-12-01T17:03:49.928 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:49.928+0000 7fbf537fe640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbf4006c790 0x7fbf4006ec50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009c00 tx=0x7fbf44009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:50.146 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:03:50.132+0000 13258640 -1 osd.0 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:50.189 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 1 -- 172.21.15.67:0/625748571 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3319+0+0 (secure 0 0 0) 0x7fbf48058740 con 0x7fbf540a8ce0 2023-12-01T17:03:50.189 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 10 client.4214.objecter ms_dispatch 0x7fbf54000b90 osd_map(14..14 src has 1..14) v4 2023-12-01T17:03:50.189 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 3 client.4214.objecter handle_osd_map got epochs [14,14] > 0 2023-12-01T17:03:50.189 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 3 client.4214.objecter handle_osd_map decoding full epoch 14 2023-12-01T17:03:50.189 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 20 client.4214.objecter dump_active .. 0 homeless 2023-12-01T17:03:50.190 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 1 -- 172.21.15.67:0/625748571 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf48054a80 con 0x7fbf540a8ce0 2023-12-01T17:03:50.190 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:50.190 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.189+0000 7fbf3bfff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:50.378 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:03:50.360+0000 13a59640 -1 osd.3 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:50.476 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.476+0000 7fbf59e56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:50.476 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.476+0000 7fbf59e56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:50.476 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.476+0000 7fbf59e56640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:50.477 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.476+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf5406b320 con 0x7fbf540a8ce0 2023-12-01T17:03:50.573 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.573+0000 7fbf3bfff640 1 -- 172.21.15.67:0/625748571 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+9518 (secure 0 0 0) 0x7fbf48021090 con 0x7fbf540a8ce0 2023-12-01T17:03:50.574 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.573+0000 7fbf3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:50.574 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.573+0000 7fbf3bfff640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:50.574 INFO:teuthology.misc.health.smithi067.stdout: 2023-12-01T17:03:50.574 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":15,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:50.431700+0000","last_up_change":"2023-12-01T17:03:50.431700+0000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6811","nonce":358761697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6813","nonce":358761697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6817","nonce":358761697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6815","nonce":358761697}]},"public_addr":"172.21.15.67:6811/358761697","cluster_addr":"172.21.15.67:6813/358761697","heartbeat_back_addr":"172.21.15.67:6817/358761697","heartbeat_front_addr":"172.21.15.67:6815/358761697","state":["exists","up"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6817","nonce":2070368587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6819","nonce":2070368587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6823","nonce":2070368587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6821","nonce":2070368587}]},"public_addr":"172.21.15.92:6817/2070368587","cluster_addr":"172.21.15.92:6819/2070368587","heartbeat_back_addr":"172.21.15.92:6823/2070368587","heartbeat_front_addr":"172.21.15.92:6821/2070368587","state":["exists","up"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6801","nonce":1636405563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6803","nonce":1636405563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6807","nonce":1636405563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6805","nonce":1636405563}]},"public_addr":"172.21.15.92:6801/1636405563","cluster_addr":"172.21.15.92:6803/1636405563","heartbeat_back_addr":"172.21.15.92:6807/1636405563","heartbeat_front_addr":"172.21.15.92:6805/1636405563","state":["exists","up"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6825","nonce":162801920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6827","nonce":162801920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6831","nonce":162801920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6829","nonce":162801920}]},"public_addr":"172.21.15.92:6825/162801920","cluster_addr":"172.21.15.92:6827/162801920","heartbeat_back_addr":"172.21.15.92:6831/162801920","heartbeat_front_addr":"172.21.15.92:6829/162801920","state":["exists","up"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6809","nonce":2695530702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6811","nonce":2695530702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6815","nonce":2695530702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6813","nonce":2695530702}]},"public_addr":"172.21.15.92:6809/2695530702","cluster_addr":"172.21.15.92:6811/2695530702","heartbeat_back_addr":"172.21.15.92:6815/2695530702","heartbeat_front_addr":"172.21.15.92:6813/2695530702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.308141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.751004+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.105081+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.199584+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:50.576 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 10 librados: watch_flush enter 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 10 librados: watch_flush exit 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 20 client.4214.objecter shutdown clearing up homeless session... 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 10 client.4214.objecter successfully canceled tick 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbf4006c790 msgr2=0x7fbf4006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbf4006c790 0x7fbf4006ec50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbf44009c00 tx=0x7fbf44009190 comp rx=0 tx=0).stop 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 10 monclient: shutdown 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8ce0 msgr2=0x7fbf541d77d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8ce0 0x7fbf541d77d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbf4800e840 tx=0x7fbf4800ed00 comp rx=0 tx=0).stop 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 shutdown_connections 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbf540a9b80 0x7fbf541da640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:50.577 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbf4006c790 0x7fbf4006ec50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:50.578 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbf540a8ce0 0x7fbf541d77d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:50.578 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 --2- 172.21.15.67:0/625748571 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbf540a8900 0x7fbf540ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:50.578 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 >> 172.21.15.67:0/625748571 conn(0x7fbf54066f00 msgr2=0x7fbf5413eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:50.578 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 shutdown_connections 2023-12-01T17:03:50.578 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 -- 172.21.15.67:0/625748571 wait complete. 2023-12-01T17:03:50.578 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:50.576+0000 7fbf59e56640 1 librados: shutdown 2023-12-01T17:03:50.589 DEBUG:teuthology.misc:5 of 8 OSDs are up 2023-12-01T17:03:50.805 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:03:50.799+0000 c493640 -1 osd.2 0 waiting for initial osdmap 2023-12-01T17:03:50.867 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:03:50.860+0000 c493640 -1 osd.1 0 waiting for initial osdmap 2023-12-01T17:03:51.250 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:03:51.236+0000 12a57640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:51.301 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:03:51.287+0000 12256640 -1 osd.1 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-01T17:03:57.396 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-12-01T17:03:57.492 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.492+0000 7f5e7f056640 10 monclient: get_monmap_and_config 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.492+0000 7f5e7f056640 10 monclient: build_initial_monmap 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.492+0000 7f5e7f056640 10 monclient: monmap: 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:03:57.493101+0000 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:03:57.493101+0000 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:57.493 INFO:teuthology.misc.health.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:57.494 INFO:teuthology.misc.health.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:57.494 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:57.495 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 1 Processor -- start 2023-12-01T17:03:57.495 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 1 -- start start 2023-12-01T17:03:57.495 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient: init 2023-12-01T17:03:57.495 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:57.495 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a8900 0x7f5e7813efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient(hunting): picked mon.noname-b con 0x7f5e780a8900 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f5e780a9b80 0x7f5e7813f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient(hunting): picked mon.noname-c con 0x7f5e780a9b80 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e780a8ce0 addr v1:172.21.15.92:6789/0 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e780b1b80 con 0x7f5e780a8ce0 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e7813fa50 con 0x7f5e780a9b80 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e780aa870 con 0x7f5e780a8900 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.493+0000 7f5e7f056640 10 monclient(hunting): authenticate will time out at 1368.731264s 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.494+0000 7f5e7cdf2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a8900 0x7f5e7813efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:57.496 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.495+0000 7f5e7cdf2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a8900 0x7f5e7813efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:37864/0 (socket says 172.21.15.67:37864) 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.495+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/4185449620 learned_addr learned my addr 172.21.15.67:0/4185449620 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.495+0000 7f5e7cdf2640 10 monclient(hunting): get_auth_request con 0x7f5e780a8900 auth_method 0 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.495+0000 7f5e7cdf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.495+0000 7f5e7cdf2640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.495+0000 7f5e7cdf2640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.496+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.496+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:57.497 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.496+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:57.498 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2023-12-01T17:03:57.498 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/4185449620 >> v2:172.21.15.92:3300/0 conn(0x7f5e780a9b80 msgr2=0x7f5e7813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:57.498 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 1 --2- 172.21.15.67:0/4185449620 >> v2:172.21.15.92:3300/0 conn(0x7f5e780a9b80 0x7f5e7813f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:57.498 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/4185449620 >> v1:172.21.15.92:6789/0 conn(0x7f5e780a8ce0 legacy=0x7f5e781453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:57.498 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 10 monclient: _finish_hunting 0 2023-12-01T17:03:57.498 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 1 monclient: found mon.noname-b 2023-12-01T17:03:57.499 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:57.499 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:03:57.499 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.498+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/4185449620 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e780aab50 con 0x7f5e780a8900 2023-12-01T17:03:57.504 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.504+0000 7f5e7cdf2640 1 --2- 172.21.15.67:0/4185449620 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a8900 0x7f5e7813efd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009870 tx=0x7f5e6c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=798e6ab003c4abce server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:57.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 1 -- 172.21.15.67:0/4185449620 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c031640 con 0x7f5e780a8900 2023-12-01T17:03:57.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:57.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:57.509 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: dump: 2023-12-01T17:03:57.509 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:57.509 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: _finish_auth 0 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: _check_auth_tickets 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 1 -- 172.21.15.67:0/4185449620 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c0317a0 con 0x7f5e780a8900 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:57.510 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 1 -- 172.21.15.67:0/4185449620 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c031b30 con 0x7f5e780a8900 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e777fe640 10 monclient: dump: 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e7f056640 5 monclient: authenticate success, global_id 4220 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e7f056640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:57.511 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.509+0000 7f5e7f056640 4 set_mon_vals no callback set 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 10 monclient: shutdown 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 -- 172.21.15.67:0/4185449620 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a8900 msgr2=0x7f5e7813efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 --2- 172.21.15.67:0/4185449620 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a8900 0x7f5e7813efd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009870 tx=0x7f5e6c02b8f0 comp rx=0 tx=0).stop 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 -- 172.21.15.67:0/4185449620 shutdown_connections 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 --2- 172.21.15.67:0/4185449620 >> v2:172.21.15.92:3300/0 conn(0x7f5e780a9b80 0x7f5e7813f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 --2- 172.21.15.67:0/4185449620 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a8900 0x7f5e7813efd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 -- 172.21.15.67:0/4185449620 >> 172.21.15.67:0/4185449620 conn(0x7f5e78066f00 msgr2=0x7f5e780b4200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 -- 172.21.15.67:0/4185449620 shutdown_connections 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 1 -- 172.21.15.67:0/4185449620 wait complete. 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 10 monclient: build_initial_monmap 2023-12-01T17:03:57.515 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.510+0000 7f5e7f056640 10 monclient: monmap: 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:epoch 0 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:created 0.000000 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 librados: starting msgr at 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 librados: starting objecter 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 Processor -- start 2023-12-01T17:03:57.516 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 -- start start 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 librados: setting wanted keys 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 librados: calling monclient init 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 10 monclient: init 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e780a8900 0x7f5e78146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 10 monclient(hunting): picked mon.noname-b con 0x7f5e780a8900 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e780a8ce0 0x7f5e781d75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.511+0000 7f5e7f056640 10 monclient(hunting): picked mon.noname-a con 0x7f5e780a8ce0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a9b80 0x7f5e781d7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 10 monclient(hunting): picked mon.noname-c con 0x7f5e780a9b80 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e781484a0 con 0x7f5e780a8ce0 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:57.517 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e78146e70 con 0x7f5e780a8900 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e781d8070 con 0x7f5e780a9b80 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7f056640 10 monclient(hunting): authenticate will time out at 1368.749496s 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7cdf2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e780a8900 0x7f5e78146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7d5f3640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a9b80 0x7f5e781d7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7cdf2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e780a8900 0x7f5e78146930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:35652/0 (socket says 172.21.15.67:35652) 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/680966121 learned_addr learned my addr 172.21.15.67:0/680966121 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7cdf2640 10 monclient(hunting): get_auth_request con 0x7f5e780a8900 auth_method 0 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7cdf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7cdf2640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7cdf2640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7d5f3640 10 monclient(hunting): get_auth_request con 0x7f5e780a9b80 auth_method 0 2023-12-01T17:03:57.518 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7d5f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7d5f3640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.512+0000 7f5e7d5f3640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.513+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.513+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.513+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.513+0000 7f5e7d5f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.513+0000 7f5e7d5f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:57.519 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.513+0000 7f5e7d5f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 10 monclient(hunting): handle_auth_done global_id 4267 payload 824 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a9b80 msgr2=0x7f5e781d7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a9b80 0x7f5e781d7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e780a8ce0 msgr2=0x7f5e781d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e780a8ce0 0x7f5e781d75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 10 monclient: _finish_hunting 0 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 1 monclient: found mon.noname-b 2023-12-01T17:03:57.520 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:57.521 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:57.521 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.515+0000 7f5e7cdf2640 1 -- 172.21.15.67:0/680966121 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c0094d0 con 0x7f5e780a8900 2023-12-01T17:03:57.521 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.517+0000 7f5e7cdf2640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e780a8900 0x7f5e78146930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009870 tx=0x7f5e6c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:57.521 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.517+0000 7f5e757fa640 10 client.?.objecter ms_handle_connect 0x7f5e780a8900 2023-12-01T17:03:57.521 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.517+0000 7f5e757fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:57.522 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 1 -- 172.21.15.67:0/680966121 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c03ac60 con 0x7f5e780a8900 2023-12-01T17:03:57.522 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:57.522 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: dump: 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: _finish_auth 0 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: _check_auth_tickets 2023-12-01T17:03:57.523 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 1 -- 172.21.15.67:0/680966121 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6c03adc0 con 0x7f5e780a8900 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 5 monclient: authenticate success, global_id 4267 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 10 monclient: _renew_subs 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e781474f0 con 0x7f5e780a8900 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 1 -- 172.21.15.67:0/680966121 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6c044630 con 0x7f5e780a8900 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e757fa640 10 monclient: dump: 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:epoch 1 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.524 INFO:teuthology.misc.health.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr: 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e76ffd640 4 set_mon_vals no callback set 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 10 monclient: _renew_subs 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e780b8320 con 0x7f5e780a8900 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.521+0000 7f5e7f056640 1 librados: init done 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.522+0000 7f5e7f056640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.522+0000 7f5e7f056640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.522+0000 7f5e7f056640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:57.525 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.522+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e78147030 con 0x7f5e780a8900 2023-12-01T17:03:57.539 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.539+0000 7f5e757fa640 1 -- 172.21.15.67:0/680966121 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7f5e6c02f050 con 0x7f5e780a8900 2023-12-01T17:03:57.539 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.539+0000 7f5e757fa640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5e5c06c7e0 0x7f5e5c06eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:57.539 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.539+0000 7f5e77fff640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5e5c06c7e0 0x7f5e5c06eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:57.539 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.539+0000 7f5e77fff640 10 monclient: get_auth_request con 0x7f5e5c06c7e0 auth_method 0 2023-12-01T17:03:57.540 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.540+0000 7f5e757fa640 10 client.4267.objecter ms_handle_connect 0x7f5e5c06c7e0 2023-12-01T17:03:57.540 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.540+0000 7f5e77fff640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5e5c06c7e0 0x7f5e5c06eca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5e78147e10 tx=0x7f5e64006bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:57.542 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.542+0000 7f5e757fa640 1 -- 172.21.15.67:0/680966121 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5e6c07ae50 con 0x7f5e780a8900 2023-12-01T17:03:57.542 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.542+0000 7f5e757fa640 10 client.4267.objecter ms_dispatch 0x7f5e78000b90 osd_map(19..19 src has 1..19) v4 2023-12-01T17:03:57.542 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.542+0000 7f5e757fa640 3 client.4267.objecter handle_osd_map got epochs [19,19] > 0 2023-12-01T17:03:57.542 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.542+0000 7f5e757fa640 3 client.4267.objecter handle_osd_map decoding full epoch 19 2023-12-01T17:03:57.542 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.542+0000 7f5e757fa640 20 client.4267.objecter dump_active .. 0 homeless 2023-12-01T17:03:57.779 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.778+0000 7f5e757fa640 1 -- 172.21.15.67:0/680966121 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5e6c076a90 con 0x7f5e780a8900 2023-12-01T17:03:57.779 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.778+0000 7f5e757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:57.779 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.778+0000 7f5e757fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:57.995 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.995+0000 7f5e7f056640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:57.995 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.995+0000 7f5e7f056640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:57.995 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.995+0000 7f5e7f056640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:57.995 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:57.995+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e780b8510 con 0x7f5e780a8900 2023-12-01T17:03:58.084 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.083+0000 7f5e757fa640 1 -- 172.21.15.67:0/680966121 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+12779 (secure 0 0 0) 0x7f5e6c031e60 con 0x7f5e780a8900 2023-12-01T17:03:58.084 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.083+0000 7f5e757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:03:58.084 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.083+0000 7f5e757fa640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:03:58.084 INFO:teuthology.misc.health.smithi067.stdout: 2023-12-01T17:03:58.084 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":19,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:03:56.008163+0000","last_up_change":"2023-12-01T17:03:52.677540+0000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-01T17:03:50.595091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6811","nonce":358761697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6813","nonce":358761697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6817","nonce":358761697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6815","nonce":358761697}]},"public_addr":"172.21.15.67:6811/358761697","cluster_addr":"172.21.15.67:6813/358761697","heartbeat_back_addr":"172.21.15.67:6817/358761697","heartbeat_front_addr":"172.21.15.67:6815/358761697","state":["exists","up"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6827","nonce":866081726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6829","nonce":866081726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6833","nonce":866081726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6831","nonce":866081726}]},"public_addr":"172.21.15.67:6827/866081726","cluster_addr":"172.21.15.67:6829/866081726","heartbeat_back_addr":"172.21.15.67:6833/866081726","heartbeat_front_addr":"172.21.15.67:6831/866081726","state":["exists","up"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6819","nonce":2342127620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6821","nonce":2342127620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6825","nonce":2342127620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6823","nonce":2342127620}]},"public_addr":"172.21.15.67:6819/2342127620","cluster_addr":"172.21.15.67:6821/2342127620","heartbeat_back_addr":"172.21.15.67:6825/2342127620","heartbeat_front_addr":"172.21.15.67:6823/2342127620","state":["exists","up"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6803","nonce":1625009733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6805","nonce":1625009733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6809","nonce":1625009733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6807","nonce":1625009733}]},"public_addr":"172.21.15.67:6803/1625009733","cluster_addr":"172.21.15.67:6805/1625009733","heartbeat_back_addr":"172.21.15.67:6809/1625009733","heartbeat_front_addr":"172.21.15.67:6807/1625009733","state":["exists","up"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6817","nonce":2070368587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6819","nonce":2070368587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6823","nonce":2070368587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6821","nonce":2070368587}]},"public_addr":"172.21.15.92:6817/2070368587","cluster_addr":"172.21.15.92:6819/2070368587","heartbeat_back_addr":"172.21.15.92:6823/2070368587","heartbeat_front_addr":"172.21.15.92:6821/2070368587","state":["exists","up"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6801","nonce":1636405563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6803","nonce":1636405563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6807","nonce":1636405563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6805","nonce":1636405563}]},"public_addr":"172.21.15.92:6801/1636405563","cluster_addr":"172.21.15.92:6803/1636405563","heartbeat_back_addr":"172.21.15.92:6807/1636405563","heartbeat_front_addr":"172.21.15.92:6805/1636405563","state":["exists","up"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6825","nonce":162801920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6827","nonce":162801920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6831","nonce":162801920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6829","nonce":162801920}]},"public_addr":"172.21.15.92:6825/162801920","cluster_addr":"172.21.15.92:6827/162801920","heartbeat_back_addr":"172.21.15.92:6831/162801920","heartbeat_front_addr":"172.21.15.92:6829/162801920","state":["exists","up"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6809","nonce":2695530702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6811","nonce":2695530702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6815","nonce":2695530702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6813","nonce":2695530702}]},"public_addr":"172.21.15.92:6809/2695530702","cluster_addr":"172.21.15.92:6811/2695530702","heartbeat_back_addr":"172.21.15.92:6815/2695530702","heartbeat_front_addr":"172.21.15.92:6813/2695530702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.603058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.949751+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.632542+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.910362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.308141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.751004+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.105081+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.199584+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 10 librados: watch_flush enter 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 10 librados: watch_flush exit 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 20 client.4267.objecter shutdown clearing up homeless session... 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 10 client.4267.objecter successfully canceled tick 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5e5c06c7e0 msgr2=0x7f5e5c06eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5e5c06c7e0 0x7f5e5c06eca0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5e78147e10 tx=0x7f5e64006bf0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 10 monclient: shutdown 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e780a8900 msgr2=0x7f5e78146930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e780a8900 0x7f5e78146930 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c009870 tx=0x7f5e6c0046c0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 shutdown_connections 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5e5c06c7e0 0x7f5e5c06eca0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5e780a9b80 0x7f5e781d7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.087 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5e780a8ce0 0x7f5e781d75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.088 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 --2- 172.21.15.67:0/680966121 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5e780a8900 0x7f5e78146930 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.088 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 >> 172.21.15.67:0/680966121 conn(0x7f5e78066f00 msgr2=0x7f5e7813d9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:58.088 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 shutdown_connections 2023-12-01T17:03:58.088 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 -- 172.21.15.67:0/680966121 wait complete. 2023-12-01T17:03:58.088 INFO:teuthology.misc.health.smithi067.stderr:2023-12-01T17:03:58.086+0000 7f5e7f056640 1 librados: shutdown 2023-12-01T17:03:58.099 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-12-01T17:03:58.099 INFO:tasks.ceph:Creating RBD pool 2023-12-01T17:03:58.100 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool create rbd 8 2023-12-01T17:03:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.220+0000 7fccd2e32640 10 monclient: get_monmap_and_config 2023-12-01T17:03:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.220+0000 7fccd2e32640 10 monclient: build_initial_monmap 2023-12-01T17:03:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.220+0000 7fccd2e32640 10 monclient: monmap: 2023-12-01T17:03:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:03:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:58.220 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:03:58.220962+0000 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:03:58.220962+0000 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:58.221 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:58.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.220+0000 7fccd2e32640 1 Processor -- start 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 1 -- start start 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient: init 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcccc0a8930 0x7fcccc0a8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient(hunting): picked mon.noname-c con 0x7fcccc0a8930 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcccc0b21a0 0x7fcccc1424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient(hunting): picked mon.noname-a con 0x7fcccc0b21a0 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcccc1429f0 addr v1:172.21.15.67:6789/0 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcccc0aa980 con 0x7fcccc1429f0 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0aabf0 con 0x7fcccc0b21a0 2023-12-01T17:03:58.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcccc0aaed0 con 0x7fcccc0a8930 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.221+0000 7fccd2e32640 10 monclient(hunting): authenticate will time out at 1369.459056s 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcccc0a8930 0x7fcccc0a8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fcccc0a8930 0x7fcccc0a8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:35662/0 (socket says 172.21.15.67:35662) 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 1 -- 172.21.15.67:0/2806227582 learned_addr learned my addr 172.21.15.67:0/2806227582 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 10 monclient(hunting): get_auth_request con 0x7fcccc0a8930 auth_method 0 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:58.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.222+0000 7fccd0bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 1 -- 172.21.15.67:0/2806227582 >> v2:172.21.15.67:3300/0 conn(0x7fcccc0b21a0 msgr2=0x7fcccc1424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 1 --2- 172.21.15.67:0/2806227582 >> v2:172.21.15.67:3300/0 conn(0x7fcccc0b21a0 0x7fcccc1424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 1 -- 172.21.15.67:0/2806227582 >> v1:172.21.15.67:6789/0 conn(0x7fcccc1429f0 legacy=0x7fcccc13f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 10 monclient: _finish_hunting 0 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 1 monclient: found mon.noname-c 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:03:58.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.224+0000 7fccd0bce640 1 -- 172.21.15.67:0/2806227582 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc141f90 con 0x7fcccc0a8930 2023-12-01T17:03:58.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.226+0000 7fccd0bce640 1 --2- 172.21.15.67:0/2806227582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0a8930 0x7fcccc0a8d10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009870 tx=0x7fccc002b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6c6f5f125617b94 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 1 -- 172.21.15.67:0/2806227582 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc00317e0 con 0x7fcccc0a8930 2023-12-01T17:03:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 10 monclient: dump: 2023-12-01T17:03:58.230 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 10 monclient: _finish_auth 0 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fcccb7fe640 1 -- 172.21.15.67:0/2806227582 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0031940 con 0x7fcccc0a8930 2023-12-01T17:03:58.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.229+0000 7fccd2e32640 5 monclient: authenticate success, global_id 4270 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fccd2e32640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fcccb7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fccd2e32640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fccd2e32640 4 set_mon_vals no callback set 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fcccb7fe640 1 -- 172.21.15.67:0/2806227582 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0031cd0 con 0x7fcccc0a8930 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fcccb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fcccb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fcccb7fe640 10 monclient: dump: 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:58.232 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:58.233 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:58.233 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fccd2e32640 10 monclient: shutdown 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fccd2e32640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fccd2e32640 1 -- 172.21.15.67:0/2806227582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0a8930 msgr2=0x7fcccc0a8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.230+0000 7fccd2e32640 1 --2- 172.21.15.67:0/2806227582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0a8930 0x7fcccc0a8d10 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fccc0009870 tx=0x7fccc002b880 comp rx=0 tx=0).stop 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 1 -- 172.21.15.67:0/2806227582 shutdown_connections 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 1 --2- 172.21.15.67:0/2806227582 >> v2:172.21.15.67:3300/0 conn(0x7fcccc0b21a0 0x7fcccc1424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 1 --2- 172.21.15.67:0/2806227582 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0a8930 0x7fcccc0a8d10 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 1 -- 172.21.15.67:0/2806227582 >> 172.21.15.67:0/2806227582 conn(0x7fcccc0b82f0 msgr2=0x7fcccc0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:58.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 1 -- 172.21.15.67:0/2806227582 shutdown_connections 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 1 -- 172.21.15.67:0/2806227582 wait complete. 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 10 monclient: build_initial_monmap 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.231+0000 7fccd2e32640 10 monclient: monmap: 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.232+0000 7fccd2e32640 1 librados: starting msgr at 2023-12-01T17:03:58.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.232+0000 7fccd2e32640 1 librados: starting objecter 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.232+0000 7fccd2e32640 1 Processor -- start 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.232+0000 7fccd2e32640 1 -- start start 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.232+0000 7fccd2e32640 1 librados: setting wanted keys 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.232+0000 7fccd2e32640 1 librados: calling monclient init 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.232+0000 7fccd2e32640 10 monclient: init 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcccc0a8930 0x7fcccc137f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): picked mon.noname-c con 0x7fcccc0a8930 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0b21a0 0x7fcccc1daa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): picked mon.noname-b con 0x7fcccc0b21a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcccc1429f0 0x7fcccc1daf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): picked mon.noname-a con 0x7fcccc1429f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:58.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc146210 con 0x7fcccc1429f0 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc1db4c0 con 0x7fcccc0b21a0 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcccc139b40 con 0x7fcccc0a8930 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fccd2e32640 10 monclient(hunting): authenticate will time out at 1369.470885s 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.233+0000 7fcccbfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0b21a0 0x7fcccc1daa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0b21a0 0x7fcccc1daa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:35668/0 (socket says 172.21.15.67:35668) 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 1 -- 172.21.15.67:0/1966277252 learned_addr learned my addr 172.21.15.67:0/1966277252 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 10 monclient(hunting): get_auth_request con 0x7fcccc0b21a0 auth_method 0 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fccd0bce640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcccc0a8930 0x7fcccc137f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:58.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fcccbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.234+0000 7fccd13cf640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcccc1429f0 0x7fcccc1daf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd13cf640 10 monclient(hunting): get_auth_request con 0x7fcccc1429f0 auth_method 0 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd13cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd13cf640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd13cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd0bce640 10 monclient(hunting): get_auth_request con 0x7fcccc0a8930 auth_method 0 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd0bce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd0bce640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd0bce640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd0bce640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd0bce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:58.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.235+0000 7fccd0bce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 10 monclient(hunting): handle_auth_done global_id 4273 payload 824 2023-12-01T17:03:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 1 -- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcccc0a8930 msgr2=0x7fcccc137f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcccc0a8930 0x7fcccc137f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 1 -- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcccc1429f0 msgr2=0x7fcccc1daf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcccc1429f0 0x7fcccc1daf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:58.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 10 monclient: _finish_hunting 0 2023-12-01T17:03:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 1 monclient: found mon.noname-b 2023-12-01T17:03:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:03:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.237+0000 7fcccbfff640 1 -- 172.21.15.67:0/1966277252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00094d0 con 0x7fcccc0b21a0 2023-12-01T17:03:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.239+0000 7fcccbfff640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0b21a0 0x7fcccc1daa40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccb800b4a0 tx=0x7fccb800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.239+0000 7fccc97fa640 10 client.?.objecter ms_handle_connect 0x7fcccc0b21a0 2023-12-01T17:03:58.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.239+0000 7fccc97fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8010610 con 0x7fcccc0b21a0 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: dump: 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:58.243 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: _finish_auth 0 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: _check_auth_tickets 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb8010770 con 0x7fcccc0b21a0 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8010b20 con 0x7fcccc0b21a0 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccc97fa640 10 monclient: dump: 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:58.244 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 5 monclient: authenticate success, global_id 4273 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 10 monclient: _renew_subs 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcccc139e20 con 0x7fcccc0b21a0 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fcccaffd640 4 set_mon_vals no callback set 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 10 monclient: _renew_subs 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:58.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcccc145da0 con 0x7fcccc0b21a0 2023-12-01T17:03:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.242+0000 7fccd2e32640 1 librados: init done 2023-12-01T17:03:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.243+0000 7fccd2e32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.243+0000 7fccd2e32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.243+0000 7fccd2e32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:58.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.243+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc04e840 con 0x7fcccc0b21a0 2023-12-01T17:03:58.481 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.480+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 89175+0+0 (secure 0 0 0) 0x7fccb8010c80 con 0x7fcccc0b21a0 2023-12-01T17:03:58.481 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.480+0000 7fccc97fa640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fccb006c790 0x7fccb006ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:58.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.480+0000 7fccd0bce640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fccb006c790 0x7fccb006ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:58.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.480+0000 7fccd0bce640 10 monclient: get_auth_request con 0x7fccb006c790 auth_method 0 2023-12-01T17:03:58.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.481+0000 7fccc97fa640 10 client.4273.objecter ms_handle_connect 0x7fccb006c790 2023-12-01T17:03:58.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.481+0000 7fccd0bce640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fccb006c790 0x7fccb006ec50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036000 tx=0x7fccc003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:58.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.483+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fccb8058400 con 0x7fcccc0b21a0 2023-12-01T17:03:58.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.483+0000 7fccc97fa640 10 client.4273.objecter ms_dispatch 0x7fcccc000b90 osd_map(19..19 src has 1..19) v4 2023-12-01T17:03:58.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.483+0000 7fccc97fa640 3 client.4273.objecter handle_osd_map got epochs [19,19] > 0 2023-12-01T17:03:58.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.483+0000 7fccc97fa640 3 client.4273.objecter handle_osd_map decoding full epoch 19 2023-12-01T17:03:58.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.483+0000 7fccc97fa640 20 client.4273.objecter dump_active .. 0 homeless 2023-12-01T17:03:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.496+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccb8054740 con 0x7fcccc0b21a0 2023-12-01T17:03:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.496+0000 7fccc97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:58.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.496+0000 7fccc97fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:03:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.675+0000 7fccd2e32640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-01T17:03:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.675+0000 7fccd2e32640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-01T17:03:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.675+0000 7fccd2e32640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:03:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.675+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fcccc138530 con 0x7fcccc0b21a0 2023-12-01T17:03:58.759 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:58.758+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fccb8054b10 con 0x7fcccc0b21a0 2023-12-01T17:03:59.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.652+0000 7fccc97fa640 1 -- 172.21.15.67:0/1966277252 <== mon.1 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v20) v1 ==== 109+0+0 (secure 0 0 0) 0x7fccb80767c0 con 0x7fcccc0b21a0 2023-12-01T17:03:59.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.652+0000 7fccc97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-12-01T17:03:59.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.652+0000 7fccc97fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-12-01T17:03:59.653 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'rbd' created 2023-12-01T17:03:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 10 librados: watch_flush enter 2023-12-01T17:03:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 10 librados: watch_flush exit 2023-12-01T17:03:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 20 client.4273.objecter shutdown clearing up homeless session... 2023-12-01T17:03:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 10 client.4273.objecter successfully canceled tick 2023-12-01T17:03:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fccb006c790 msgr2=0x7fccb006ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:59.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fccb006c790 0x7fccb006ec50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036000 tx=0x7fccc003d040 comp rx=0 tx=0).stop 2023-12-01T17:03:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 10 monclient: shutdown 2023-12-01T17:03:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0b21a0 msgr2=0x7fcccc1daa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:59.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0b21a0 0x7fcccc1daa40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fccb800b4a0 tx=0x7fccb800b960 comp rx=0 tx=0).stop 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 shutdown_connections 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcccc1429f0 0x7fcccc1daf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcccc0b21a0 0x7fcccc1daa40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fccb006c790 0x7fccb006ec50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 --2- 172.21.15.67:0/1966277252 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcccc0a8930 0x7fcccc137f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 >> 172.21.15.67:0/1966277252 conn(0x7fcccc0b82f0 msgr2=0x7fcccc0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 shutdown_connections 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 -- 172.21.15.67:0/1966277252 wait complete. 2023-12-01T17:03:59.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.654+0000 7fccd2e32640 1 librados: shutdown 2023-12-01T17:03:59.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-01T17:03:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.817+0000 7f2e7c234640 10 monclient: get_monmap_and_config 2023-12-01T17:03:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.817+0000 7f2e7c234640 10 monclient: build_initial_monmap 2023-12-01T17:03:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.817+0000 7f2e7c234640 10 monclient: monmap: 2023-12-01T17:03:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:03:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:03:59.818779+0000 2023-12-01T17:03:59.818 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:03:59.818779+0000 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:03:59.819 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.817+0000 7f2e7c234640 1 Processor -- start 2023-12-01T17:03:59.821 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.817+0000 7f2e7c234640 1 -- start start 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.817+0000 7f2e7c234640 10 monclient: init 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e740a8990 0x7f2e740a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): picked mon.noname-a con 0x7f2e740a8990 addr v2:172.21.15.67:3300/0 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e740b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): picked mon.noname-b con 0x7f2e741427b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2e7413ef10 0x7f2e7413f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): picked mon.noname-c con 0x7f2e7413ef10 addr v2:172.21.15.92:3300/0 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e740aabe0 con 0x7f2e740a8990 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e740aaec0 con 0x7f2e7413ef10 2023-12-01T17:03:59.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74142260 con 0x7f2e741427b0 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e7c234640 10 monclient(hunting): authenticate will time out at 1371.056861s 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.818+0000 7f2e797cf640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e740b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e79fd0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f2e740a8990 0x7f2e740a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e7a7d1640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2e7413ef10 0x7f2e7413f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e740b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:37902/0 (socket says 172.21.15.67:37902) 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 1 -- 172.21.15.67:0/4125380453 learned_addr learned my addr 172.21.15.67:0/4125380453 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 10 monclient(hunting): get_auth_request con 0x7f2e741427b0 auth_method 0 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:59.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e79fd0640 10 monclient(hunting): get_auth_request con 0x7f2e740a8990 auth_method 0 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e79fd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e79fd0640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e79fd0640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e7a7d1640 10 monclient(hunting): get_auth_request con 0x7f2e7413ef10 auth_method 0 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e7a7d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e7a7d1640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e7a7d1640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.819+0000 7f2e797cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.820+0000 7f2e79fd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.820+0000 7f2e79fd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:59.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.820+0000 7f2e79fd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.820+0000 7f2e7a7d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.820+0000 7f2e7a7d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.820+0000 7f2e7a7d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2023-12-01T17:03:59.825 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 1 -- 172.21.15.67:0/4125380453 >> v2:172.21.15.92:3300/0 conn(0x7f2e7413ef10 msgr2=0x7f2e7413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 1 --2- 172.21.15.67:0/4125380453 >> v2:172.21.15.92:3300/0 conn(0x7f2e7413ef10 0x7f2e7413f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 1 -- 172.21.15.67:0/4125380453 >> v2:172.21.15.67:3300/0 conn(0x7f2e740a8990 msgr2=0x7f2e740a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 1 --2- 172.21.15.67:0/4125380453 >> v2:172.21.15.67:3300/0 conn(0x7f2e740a8990 0x7f2e740a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 10 monclient: _finish_hunting 0 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 1 monclient: found mon.noname-b 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.821+0000 7f2e797cf640 1 -- 172.21.15.67:0/4125380453 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e740b26e0 con 0x7f2e741427b0 2023-12-01T17:03:59.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.823+0000 7f2e797cf640 1 --2- 172.21.15.67:0/4125380453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e740b21a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e740b3d20 tx=0x7f2e6402b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6dbcc317506a37f server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e78fce640 1 -- 172.21.15.67:0/4125380453 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e640317e0 con 0x7f2e741427b0 2023-12-01T17:03:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e78fce640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:59.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e78fce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e78fce640 10 monclient: dump: 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e78fce640 10 monclient: _finish_auth 0 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e78fce640 10 monclient: _check_auth_tickets 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e78fce640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:59.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e7c234640 5 monclient: authenticate success, global_id 4232 2023-12-01T17:03:59.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.826+0000 7f2e7c234640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:03:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e78fce640 1 -- 172.21.15.67:0/4125380453 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e64031940 con 0x7f2e741427b0 2023-12-01T17:03:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e78fce640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e7c234640 10 monclient: get_monmap_and_config success 2023-12-01T17:03:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e7c234640 4 set_mon_vals no callback set 2023-12-01T17:03:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e78fce640 1 -- 172.21.15.67:0/4125380453 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e64031cd0 con 0x7f2e741427b0 2023-12-01T17:03:59.830 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e78fce640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e78fce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.827+0000 7f2e78fce640 10 monclient: dump: 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 10 monclient: shutdown 2023-12-01T17:03:59.831 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 -- 172.21.15.67:0/4125380453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 msgr2=0x7f2e740b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 --2- 172.21.15.67:0/4125380453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e740b21a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2e740b3d20 tx=0x7f2e6402b880 comp rx=0 tx=0).stop 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 -- 172.21.15.67:0/4125380453 shutdown_connections 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 --2- 172.21.15.67:0/4125380453 >> v2:172.21.15.92:3300/0 conn(0x7f2e7413ef10 0x7f2e7413f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 --2- 172.21.15.67:0/4125380453 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e740b21a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 --2- 172.21.15.67:0/4125380453 >> v2:172.21.15.67:3300/0 conn(0x7f2e740a8990 0x7f2e740a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 -- 172.21.15.67:0/4125380453 >> 172.21.15.67:0/4125380453 conn(0x7f2e740b8310 msgr2=0x7f2e740b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 -- 172.21.15.67:0/4125380453 shutdown_connections 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 1 -- 172.21.15.67:0/4125380453 wait complete. 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 10 monclient: build_initial_monmap 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.828+0000 7f2e7c234640 10 monclient: monmap: 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:03:59.832 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 librados: starting msgr at 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 librados: starting objecter 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 Processor -- start 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 -- start start 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 librados: setting wanted keys 2023-12-01T17:03:59.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 librados: calling monclient init 2023-12-01T17:03:59.834 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient: init 2023-12-01T17:03:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient: _reopen_session rank -1 2023-12-01T17:03:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:03:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e740a8990 0x7f2e740b0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): picked mon.noname-a con 0x7f2e740a8990 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:03:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e7413ef10 0x7f2e741405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): picked mon.noname-b con 0x7f2e7413ef10 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:03:59.836 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e741d71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): picked mon.noname-c con 0x7f2e741427b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74140af0 con 0x7f2e740a8990 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74141280 con 0x7f2e7413ef10 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): start opening mon connection 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e74140d60 con 0x7f2e741427b0 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): _renew_subs 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.829+0000 7f2e7c234640 10 monclient(hunting): authenticate will time out at 1371.068381s 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e7a7d1640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e741d71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e79fd0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e740a8990 0x7f2e740b0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e797cf640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e7413ef10 0x7f2e741405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e7a7d1640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e741d71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:37918/0 (socket says 172.21.15.67:37918) 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e7a7d1640 1 -- 172.21.15.67:0/641868407 learned_addr learned my addr 172.21.15.67:0/641868407 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:03:59.837 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e7a7d1640 10 monclient(hunting): get_auth_request con 0x7f2e741427b0 auth_method 0 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e7a7d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e7a7d1640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e7a7d1640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e797cf640 10 monclient(hunting): get_auth_request con 0x7f2e7413ef10 auth_method 0 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e797cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e797cf640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.830+0000 7f2e797cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e79fd0640 10 monclient(hunting): get_auth_request con 0x7f2e740a8990 auth_method 0 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e79fd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e79fd0640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e79fd0640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e7a7d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e7a7d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e7a7d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:59.838 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e797cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e797cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e797cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e79fd0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e79fd0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.831+0000 7f2e79fd0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 1 -- 172.21.15.67:0/641868407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e7413ef10 msgr2=0x7f2e741405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e7413ef10 0x7f2e741405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 1 -- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e740a8990 msgr2=0x7f2e740b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e740a8990 0x7f2e740b0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 10 monclient: _finish_hunting 0 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 1 monclient: found mon.noname-c 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:03:59.839 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.834+0000 7f2e7a7d1640 1 -- 172.21.15.67:0/641868407 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e640094d0 con 0x7f2e741427b0 2023-12-01T17:03:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.836+0000 7f2e7a7d1640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e741d71b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00efc0 tx=0x7f2e6c00c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.836+0000 7f2e6a7fc640 10 client.?.objecter ms_handle_connect 0x7f2e741427b0 2023-12-01T17:03:59.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.836+0000 7f2e6a7fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:03:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 1 -- 172.21.15.67:0/641868407 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c014480 con 0x7f2e741427b0 2023-12-01T17:03:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:03:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: dump: 2023-12-01T17:03:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:59.841 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: _finish_auth 0 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 1 -- 172.21.15.67:0/641868407 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e6c00f040 con 0x7f2e741427b0 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 1 -- 172.21.15.67:0/641868407 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6c014820 con 0x7f2e741427b0 2023-12-01T17:03:59.842 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 5 monclient: authenticate success, global_id 4235 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 monclient: _renew_subs 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7413fb80 con 0x7f2e741427b0 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6bfff640 4 set_mon_vals no callback set 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e6a7fc640 10 monclient: dump: 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:03:59.843 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:03:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:03:59.844 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:03:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:03:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 monclient: _renew_subs 2023-12-01T17:03:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e741d76f0 con 0x7f2e741427b0 2023-12-01T17:03:59.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 1 librados: init done 2023-12-01T17:03:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:03:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:03:59.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.840+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e740a97e0 con 0x7f2e741427b0 2023-12-01T17:03:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.860+0000 7f2e6a7fc640 1 -- 172.21.15.67:0/641868407 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2e6c014980 con 0x7f2e741427b0 2023-12-01T17:03:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.860+0000 7f2e6a7fc640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2e5806c870 0x7f2e5806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:03:59.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.860+0000 7f2e79fd0640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2e5806c870 0x7f2e5806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:03:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.860+0000 7f2e79fd0640 10 monclient: get_auth_request con 0x7f2e5806c870 auth_method 0 2023-12-01T17:03:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.861+0000 7f2e79fd0640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2e5806c870 0x7f2e5806ed30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009b40 tx=0x7f2e70009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:03:59.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.861+0000 7f2e6a7fc640 10 client.4235.objecter ms_handle_connect 0x7f2e5806c870 2023-12-01T17:03:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.863+0000 7f2e6a7fc640 1 -- 172.21.15.67:0/641868407 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f2e6c0595f0 con 0x7f2e741427b0 2023-12-01T17:03:59.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.863+0000 7f2e6a7fc640 10 client.4235.objecter ms_dispatch 0x7f2e74000b90 osd_map(20..20 src has 1..20) v4 2023-12-01T17:03:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.863+0000 7f2e6a7fc640 3 client.4235.objecter handle_osd_map got epochs [20,20] > 0 2023-12-01T17:03:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.863+0000 7f2e6a7fc640 3 client.4235.objecter handle_osd_map decoding full epoch 20 2023-12-01T17:03:59.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:03:59.863+0000 7f2e6a7fc640 20 client.4235.objecter dump_active .. 0 homeless 2023-12-01T17:04:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.134+0000 7f2e6a7fc640 1 -- 172.21.15.67:0/641868407 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e6c055930 con 0x7f2e741427b0 2023-12-01T17:04:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.134+0000 7f2e6a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:00.135 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.134+0000 7f2e6a7fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.326+0000 7f2e7c234640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-01T17:04:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.326+0000 7f2e7c234640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-01T17:04:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.326+0000 7f2e7c234640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:00.328 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.326+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2e74140fa0 con 0x7f2e741427b0 2023-12-01T17:04:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.843+0000 7f2e6a7fc640 1 -- 172.21.15.67:0/641868407 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v21) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2e74140fa0 con 0x7f2e741427b0 2023-12-01T17:04:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.843+0000 7f2e6a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-12-01T17:04:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.843+0000 7f2e6a7fc640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-12-01T17:04:00.845 INFO:teuthology.orchestra.run.smithi067.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 10 librados: watch_flush enter 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 10 librados: watch_flush exit 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 20 client.4235.objecter shutdown clearing up homeless session... 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 10 client.4235.objecter successfully canceled tick 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2e5806c870 msgr2=0x7f2e5806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2e5806c870 0x7f2e5806ed30 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009b40 tx=0x7f2e70009190 comp rx=0 tx=0).stop 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 10 monclient: shutdown 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 msgr2=0x7f2e741d71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:00.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e741d71b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c00efc0 tx=0x7f2e6c00c490 comp rx=0 tx=0).stop 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 shutdown_connections 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2e5806c870 0x7f2e5806ed30 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2e741427b0 0x7f2e741d71b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e7413ef10 0x7f2e741405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 --2- 172.21.15.67:0/641868407 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e740a8990 0x7f2e740b0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 >> 172.21.15.67:0/641868407 conn(0x7f2e740b8310 msgr2=0x7f2e740b2930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 shutdown_connections 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 -- 172.21.15.67:0/641868407 wait complete. 2023-12-01T17:04:00.849 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.846+0000 7f2e7c234640 1 librados: shutdown 2023-12-01T17:04:00.858 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-12-01T17:04:00.859 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-12-01T17:04:00.859 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-12-01T17:04:00.859 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-12-01T17:04:00.860 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-01T17:04:00.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.971+0000 7fed5ab3e640 10 monclient: get_monmap_and_config 2023-12-01T17:04:00.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.971+0000 7fed5ab3e640 10 monclient: build_initial_monmap 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.971+0000 7fed5ab3e640 10 monclient: monmap: 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:00.973189+0000 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:00.973189+0000 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:00.973 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 1 Processor -- start 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 1 -- start start 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient: init 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed540a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed541461c0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fed540a8e00 0x7fed5413f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient(hunting): picked mon.noname-a con 0x7fed540a8e00 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed54145620 con 0x7fed541461c0 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed540aace0 con 0x7fed540a8a20 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed540aafc0 con 0x7fed540a8e00 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:00.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.972+0000 7fed5ab3e640 10 monclient(hunting): authenticate will time out at 1372.211372s 2023-12-01T17:04:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.973+0000 7fed588da640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fed540a8a20 0x7fed54145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:49328/0 (socket says 172.21.15.67:49328) 2023-12-01T17:04:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.973+0000 7fed588da640 1 -- 172.21.15.67:0/308437723 learned_addr learned my addr 172.21.15.67:0/308437723 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:00.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.973+0000 7fed590db640 1 --2- 172.21.15.67:0/308437723 >> v2:172.21.15.67:3300/0 conn(0x7fed540a8e00 0x7fed5413f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed537fe640 1 -- 172.21.15.67:0/308437723 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538556908 0 0) 0x7fed540aace0 con 0x7fed540a8a20 2023-12-01T17:04:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed537fe640 10 monclient(hunting): _negotiate 2023-12-01T17:04:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed537fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed537fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed537fe640 10 monclient(hunting): my global_id is 4282 2023-12-01T17:04:00.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed537fe640 1 -- 172.21.15.67:0/308437723 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed44003520 con 0x7fed540a8a20 2023-12-01T17:04:00.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed590db640 10 monclient(hunting): get_auth_request con 0x7fed540a8e00 auth_method 0 2023-12-01T17:04:00.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed590db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:00.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed590db640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:00.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.976+0000 7fed590db640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:00.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.977+0000 7fed590db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:00.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.977+0000 7fed590db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:00.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.977+0000 7fed590db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 10 monclient(hunting): handle_auth_done global_id 4242 payload 274 2023-12-01T17:04:00.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 1 -- 172.21.15.67:0/308437723 >> v1:172.21.15.92:6789/0 conn(0x7fed540a8a20 legacy=0x7fed54145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 1 -- 172.21.15.67:0/308437723 >> v1:172.21.15.67:6789/0 conn(0x7fed541461c0 legacy=0x7fed5413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 10 monclient: _finish_hunting 0 2023-12-01T17:04:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 1 monclient: found mon.noname-a 2023-12-01T17:04:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:00.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.979+0000 7fed590db640 1 -- 172.21.15.67:0/308437723 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed540b2210 con 0x7fed540a8e00 2023-12-01T17:04:00.985 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.984+0000 7fed590db640 1 --2- 172.21.15.67:0/308437723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed540a8e00 0x7fed5413f800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009d90 tx=0x7fed4c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5953d6d2b92b7438 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 1 -- 172.21.15.67:0/308437723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c02cd00 con 0x7fed540a8e00 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: dump: 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:00.990 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: _finish_auth 0 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: _check_auth_tickets 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 1 -- 172.21.15.67:0/308437723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed4c02ce60 con 0x7fed540a8e00 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed5ab3e640 5 monclient: authenticate success, global_id 4242 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed5ab3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed5ab3e640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 1 -- 172.21.15.67:0/308437723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed4c034660 con 0x7fed540a8e00 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed5ab3e640 4 set_mon_vals no callback set 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:00.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.988+0000 7fed537fe640 10 monclient: dump: 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:00.992 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 10 monclient: shutdown 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 1 -- 172.21.15.67:0/308437723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed540a8e00 msgr2=0x7fed5413f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/308437723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed540a8e00 0x7fed5413f800 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fed4c009d90 tx=0x7fed4c02bf50 comp rx=0 tx=0).stop 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 1 -- 172.21.15.67:0/308437723 shutdown_connections 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/308437723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed540a8e00 0x7fed5413f800 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 1 -- 172.21.15.67:0/308437723 >> 172.21.15.67:0/308437723 conn(0x7fed540b7890 msgr2=0x7fed540b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 1 -- 172.21.15.67:0/308437723 shutdown_connections 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.989+0000 7fed5ab3e640 1 -- 172.21.15.67:0/308437723 wait complete. 2023-12-01T17:04:00.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.990+0000 7fed5ab3e640 10 monclient: build_initial_monmap 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.990+0000 7fed5ab3e640 10 monclient: monmap: 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.990+0000 7fed5ab3e640 1 librados: starting msgr at 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.990+0000 7fed5ab3e640 1 librados: starting objecter 2023-12-01T17:04:00.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 1 Processor -- start 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 1 -- start start 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 1 librados: setting wanted keys 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 1 librados: calling monclient init 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 10 monclient: init 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed540a8a20 0x7fed54142820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 10 monclient(hunting): picked mon.noname-c con 0x7fed540a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed540a8e00 0x7fed54134aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.991+0000 7fed5ab3e640 10 monclient(hunting): picked mon.noname-b con 0x7fed540a8e00 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed541461c0 0x7fed54132600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 10 monclient(hunting): picked mon.noname-a con 0x7fed541461c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54142d60 con 0x7fed541461c0 2023-12-01T17:04:00.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54142250 con 0x7fed540a8e00 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed54136170 con 0x7fed540a8a20 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed5ab3e640 10 monclient(hunting): authenticate will time out at 1372.230449s 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed588da640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed540a8a20 0x7fed54142820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed53fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed540a8e00 0x7fed54134aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.992+0000 7fed588da640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed540a8a20 0x7fed54142820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:37932/0 (socket says 172.21.15.67:37932) 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 1 -- 172.21.15.67:0/3587233576 learned_addr learned my addr 172.21.15.67:0/3587233576 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 10 monclient(hunting): get_auth_request con 0x7fed540a8a20 auth_method 0 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:00.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed53fff640 10 monclient(hunting): get_auth_request con 0x7fed540a8e00 auth_method 0 2023-12-01T17:04:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed53fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed53fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed53fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.993+0000 7fed588da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:00.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.994+0000 7fed590db640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed541461c0 0x7fed54132600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.994+0000 7fed590db640 10 monclient(hunting): get_auth_request con 0x7fed541461c0 auth_method 0 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.994+0000 7fed590db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.994+0000 7fed590db640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.994+0000 7fed590db640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.995+0000 7fed590db640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.995+0000 7fed590db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.995+0000 7fed590db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.995+0000 7fed53fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.995+0000 7fed53fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.995+0000 7fed53fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:01.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 1 -- 172.21.15.67:0/3587233576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed540a8e00 msgr2=0x7fed54134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed540a8e00 0x7fed54134aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 1 -- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed541461c0 msgr2=0x7fed54132600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed541461c0 0x7fed54132600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 10 monclient: _finish_hunting 0 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 1 monclient: found mon.noname-c 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:01.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:00.997+0000 7fed588da640 1 -- 172.21.15.67:0/3587233576 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4c009a70 con 0x7fed540a8a20 2023-12-01T17:04:01.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.000+0000 7fed588da640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed540a8a20 0x7fed54142820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed4800c530 tx=0x7fed4800c9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:01.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.000+0000 7fed517fa640 10 client.?.objecter ms_handle_connect 0x7fed540a8a20 2023-12-01T17:04:01.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.000+0000 7fed517fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 1 -- 172.21.15.67:0/3587233576 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed48005ce0 con 0x7fed540a8a20 2023-12-01T17:04:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: dump: 2023-12-01T17:04:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:01.007 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: _finish_auth 0 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: _check_auth_tickets 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 1 -- 172.21.15.67:0/3587233576 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48005e40 con 0x7fed540a8a20 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 5 monclient: authenticate success, global_id 4238 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:01.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 1 -- 172.21.15.67:0/3587233576 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed480072b0 con 0x7fed540a8a20 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed517fa640 10 monclient: dump: 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 10 monclient: _renew_subs 2023-12-01T17:04:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed52ffd640 4 set_mon_vals no callback set 2023-12-01T17:04:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed54134fe0 con 0x7fed540a8a20 2023-12-01T17:04:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 10 monclient: _renew_subs 2023-12-01T17:04:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed54136770 con 0x7fed540a8a20 2023-12-01T17:04:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.005+0000 7fed5ab3e640 1 librados: init done 2023-12-01T17:04:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.006+0000 7fed5ab3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.006+0000 7fed5ab3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.006+0000 7fed5ab3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.006+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5404e840 con 0x7fed540a8a20 2023-12-01T17:04:01.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.288+0000 7fed517fa640 1 -- 172.21.15.67:0/3587233576 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fed48007410 con 0x7fed540a8a20 2023-12-01T17:04:01.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.289+0000 7fed517fa640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fed3806c820 0x7fed3806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:01.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.300+0000 7fed53fff640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fed3806c820 0x7fed3806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:01.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.300+0000 7fed53fff640 10 monclient: get_auth_request con 0x7fed3806c820 auth_method 0 2023-12-01T17:04:01.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.300+0000 7fed53fff640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fed3806c820 0x7fed3806ece0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed5414d860 tx=0x7fed40009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:01.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.300+0000 7fed517fa640 10 client.4238.objecter ms_handle_connect 0x7fed3806c820 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 1 -- 172.21.15.67:0/3587233576 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fed48057f00 con 0x7fed540a8a20 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 10 client.4238.objecter ms_dispatch 0x7fed54000b90 osd_map(21..21 src has 1..21) v4 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 3 client.4238.objecter handle_osd_map got epochs [21,21] > 0 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 3 client.4238.objecter handle_osd_map decoding full epoch 21 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 20 client.4238.objecter dump_active .. 0 homeless 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 1 -- 172.21.15.67:0/3587233576 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed48054240 con 0x7fed540a8a20 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:01.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.307+0000 7fed517fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.595+0000 7fed5ab3e640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-12-01T17:04:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.595+0000 7fed5ab3e640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-01T17:04:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.595+0000 7fed5ab3e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:01.597 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.595+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fed54139db0 con 0x7fed540a8a20 2023-12-01T17:04:01.752 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.750+0000 7fed517fa640 1 -- 172.21.15.67:0/3587233576 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171032 (secure 0 0 0) 0x7fed48057860 con 0x7fed540a8a20 2023-12-01T17:04:01.752 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.750+0000 7fed517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-12-01T17:04:01.752 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.750+0000 7fed517fa640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:01.754 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:04:01.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 10 librados: watch_flush enter 2023-12-01T17:04:01.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 10 librados: watch_flush exit 2023-12-01T17:04:01.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 20 client.4238.objecter shutdown clearing up homeless session... 2023-12-01T17:04:01.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 10 client.4238.objecter successfully canceled tick 2023-12-01T17:04:01.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fed3806c820 msgr2=0x7fed3806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:01.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fed3806c820 0x7fed3806ece0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fed5414d860 tx=0x7fed40009040 comp rx=0 tx=0).stop 2023-12-01T17:04:01.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 10 monclient: shutdown 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed540a8a20 msgr2=0x7fed54142820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed540a8a20 0x7fed54142820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed4800c530 tx=0x7fed4800c9f0 comp rx=0 tx=0).stop 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 shutdown_connections 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fed541461c0 0x7fed54132600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fed540a8e00 0x7fed54134aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fed3806c820 0x7fed3806ece0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 --2- 172.21.15.67:0/3587233576 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fed540a8a20 0x7fed54142820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 >> 172.21.15.67:0/3587233576 conn(0x7fed540b7890 msgr2=0x7fed540b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 shutdown_connections 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 -- 172.21.15.67:0/3587233576 wait complete. 2023-12-01T17:04:01.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.753+0000 7fed5ab3e640 1 librados: shutdown 2023-12-01T17:04:01.766 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":5,"active_gid":4100,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":1043126838},{"type":"v1","addr":"172.21.15.67:6801","nonce":1043126838}]},"active_addr":"172.21.15.67:6801/1043126838","active_change":"2023-12-01T17:03:09.757171+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1654950556}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3479020390}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3283175568}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3292081298}]}]} 2023-12-01T17:04:01.769 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-12-01T17:04:01.769 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-12-01T17:04:01.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T17:04:01.867 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:15.647 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:01.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.877+0000 7f004d79e640 10 monclient: get_monmap_and_config 2023-12-01T17:04:01.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.877+0000 7f004d79e640 10 monclient: build_initial_monmap 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.877+0000 7f004d79e640 10 monclient: monmap: 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:01.879089+0000 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:01.879089+0000 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:01.879 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:01.880 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.878+0000 7f004d79e640 1 Processor -- start 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.878+0000 7f004d79e640 1 -- start start 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.878+0000 7f004d79e640 10 monclient: init 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f00480a8730 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f004813efa0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f004813f410 0x7f004814d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient(hunting): picked mon.noname-c con 0x7f004813f410 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00480aab60 con 0x7f004813efa0 2023-12-01T17:04:01.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00480aadb0 con 0x7f00480a8730 2023-12-01T17:04:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0048141eb0 con 0x7f004813f410 2023-12-01T17:04:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:01.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.879+0000 7f004d79e640 10 monclient(hunting): authenticate will time out at 1373.117686s 2023-12-01T17:04:01.922 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:15.702 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:01.922 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:15.682 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:01.934 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:15.714 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:01.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.947+0000 7f004cf9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f004813f410 0x7f004814d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.971+0000 7f0047fff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f00480a8730 0x7f00480a8b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57724/0 (socket says 172.21.15.67:57724) 2023-12-01T17:04:01.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.971+0000 7f0047fff640 1 -- 172.21.15.67:0/1125758785 learned_addr learned my addr 172.21.15.67:0/1125758785 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:01.984 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.983+0000 7f004cf9d640 10 monclient(hunting): get_auth_request con 0x7f004813f410 auth_method 0 2023-12-01T17:04:01.984 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.983+0000 7f004cf9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:01.984 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.983+0000 7f004cf9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:01.984 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:01.983+0000 7f004cf9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:01.985 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:15.744 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:01.988 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:15.769 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:02.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.049+0000 7f004cf9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:02.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.049+0000 7f004cf9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:02.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.049+0000 7f004cf9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:02.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2023-12-01T17:04:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 1 -- 172.21.15.67:0/1125758785 >> v1:172.21.15.92:6789/0 conn(0x7f00480a8730 legacy=0x7f00480a8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 1 -- 172.21.15.67:0/1125758785 >> v1:172.21.15.67:6789/0 conn(0x7f004813efa0 legacy=0x7f0048142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 1 monclient: found mon.noname-c 2023-12-01T17:04:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:02.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.055+0000 7f004cf9d640 1 -- 172.21.15.67:0/1125758785 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00480a9840 con 0x7f004813f410 2023-12-01T17:04:02.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.118+0000 7f004cf9d640 1 --2- 172.21.15.67:0/1125758785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 0x7f004814d610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0040009d90 tx=0x7f004002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10e31501956429e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:02.200 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:15.981 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:02.227 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:15.987 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:02.230 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:16.011 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:02.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 1 -- 172.21.15.67:0/1125758785 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f004002cd00 con 0x7f004813f410 2023-12-01T17:04:02.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:02.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:02.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: dump: 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: _finish_auth 0 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:02.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 1 -- 172.21.15.67:0/1125758785 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f004002ce60 con 0x7f004813f410 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 5 monclient: authenticate success, global_id 4288 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 1 -- 172.21.15.67:0/1125758785 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0040034660 con 0x7f004813f410 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 4 set_mon_vals no callback set 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f0046ffd640 10 monclient: dump: 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.248 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:02.249 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:02.249 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 10 monclient: shutdown 2023-12-01T17:04:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 1 -- 172.21.15.67:0/1125758785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 msgr2=0x7f004814d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.245+0000 7f004d79e640 1 --2- 172.21.15.67:0/1125758785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 0x7f004814d610 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0040009d90 tx=0x7f004002bf50 comp rx=0 tx=0).stop 2023-12-01T17:04:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.246+0000 7f004d79e640 1 -- 172.21.15.67:0/1125758785 shutdown_connections 2023-12-01T17:04:02.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.246+0000 7f004d79e640 1 --2- 172.21.15.67:0/1125758785 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 0x7f004814d610 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.246+0000 7f004d79e640 1 -- 172.21.15.67:0/1125758785 >> 172.21.15.67:0/1125758785 conn(0x7f00480b77a0 msgr2=0x7f00480b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.246+0000 7f004d79e640 1 -- 172.21.15.67:0/1125758785 shutdown_connections 2023-12-01T17:04:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.246+0000 7f004d79e640 1 -- 172.21.15.67:0/1125758785 wait complete. 2023-12-01T17:04:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.246+0000 7f004d79e640 10 monclient: build_initial_monmap 2023-12-01T17:04:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 10 monclient: monmap: 2023-12-01T17:04:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:02.252 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 1 librados: starting msgr at 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 1 librados: starting objecter 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 1 Processor -- start 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 1 -- start start 2023-12-01T17:04:02.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 1 librados: setting wanted keys 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 1 librados: calling monclient init 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.247+0000 7f004d79e640 10 monclient: init 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00480a8730 0x7f00480ab810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): picked mon.noname-a con 0x7f00480a8730 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004813efa0 0x7f00480abdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): picked mon.noname-c con 0x7f004813efa0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 0x7f0048143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): picked mon.noname-b con 0x7f004813f410 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00480ad990 con 0x7f00480a8730 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:02.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00480ac330 con 0x7f004813f410 2023-12-01T17:04:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00480adc70 con 0x7f004813efa0 2023-12-01T17:04:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.248+0000 7f004d79e640 10 monclient(hunting): authenticate will time out at 1373.486828s 2023-12-01T17:04:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.252+0000 7f00477fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004813efa0 0x7f00480abdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:02.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.254+0000 7f0047fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00480a8730 0x7f00480ab810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.255+0000 7f00477fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004813efa0 0x7f00480abdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41684/0 (socket says 172.21.15.67:41684) 2023-12-01T17:04:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.255+0000 7f00477fe640 1 -- 172.21.15.67:0/4123559890 learned_addr learned my addr 172.21.15.67:0/4123559890 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.255+0000 7f00477fe640 10 monclient(hunting): get_auth_request con 0x7f004813efa0 auth_method 0 2023-12-01T17:04:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.255+0000 7f00477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.255+0000 7f00477fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:02.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.255+0000 7f00477fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:02.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.259+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:02.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.259+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:02.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.259+0000 7f00477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:02.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.260+0000 7f0047fff640 10 monclient(hunting): get_auth_request con 0x7f00480a8730 auth_method 0 2023-12-01T17:04:02.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.260+0000 7f0047fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:02.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.260+0000 7f0047fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:02.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.260+0000 7f0047fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:02.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.263+0000 7f004cf9d640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 0x7f0048143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:02.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.270+0000 7f004cf9d640 10 monclient(hunting): get_auth_request con 0x7f004813f410 auth_method 0 2023-12-01T17:04:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.270+0000 7f004cf9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.270+0000 7f004cf9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:02.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.270+0000 7f004cf9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.275+0000 7f004cf9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.275+0000 7f004cf9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:02.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.275+0000 7f004cf9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.301+0000 7f0047fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.301+0000 7f0047fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:02.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.301+0000 7f0047fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2023-12-01T17:04:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 1 -- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004813efa0 msgr2=0x7f00480abdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004813efa0 0x7f00480abdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 1 -- 172.21.15.67:0/4123559890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 msgr2=0x7f0048143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:02.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 0x7f0048143ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 1 monclient: found mon.noname-a 2023-12-01T17:04:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:02.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.310+0000 7f0047fff640 1 -- 172.21.15.67:0/4123559890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0040009a70 con 0x7f00480a8730 2023-12-01T17:04:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.325+0000 7f0047fff640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00480a8730 0x7f00480ab810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f003800c440 tx=0x7f0038001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.325+0000 7f0044ff9640 10 client.?.objecter ms_handle_connect 0x7f00480a8730 2023-12-01T17:04:02.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.325+0000 7f0044ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.343+0000 7f0044ff9640 1 -- 172.21.15.67:0/4123559890 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0038005ce0 con 0x7f00480a8730 2023-12-01T17:04:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.343+0000 7f0044ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:02.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.343+0000 7f0044ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.343+0000 7f0044ff9640 10 monclient: dump: 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 10 monclient: _finish_auth 0 2023-12-01T17:04:02.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 1 -- 172.21.15.67:0/4123559890 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0038005e40 con 0x7f00480a8730 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 5 monclient: authenticate success, global_id 4251 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 10 monclient: _renew_subs 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00480ac5e0 con 0x7f00480a8730 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 1 -- 172.21.15.67:0/4123559890 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00380072d0 con 0x7f00480a8730 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f0044ff9640 10 monclient: dump: 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:02.347 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f00467fc640 4 set_mon_vals no callback set 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 10 monclient: _renew_subs 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00481440e0 con 0x7f00480a8730 2023-12-01T17:04:02.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.344+0000 7f004d79e640 1 librados: init done 2023-12-01T17:04:02.349 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.345+0000 7f004d79e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:02.349 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.345+0000 7f004d79e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:02.349 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.345+0000 7f004d79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:02.349 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.345+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0048075a40 con 0x7f00480a8730 2023-12-01T17:04:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.615+0000 7f0044ff9640 1 -- 172.21.15.67:0/4123559890 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f00380074f0 con 0x7f00480a8730 2023-12-01T17:04:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.615+0000 7f0044ff9640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f003406c870 0x7f003406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.616+0000 7f00477fe640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f003406c870 0x7f003406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.616+0000 7f00477fe640 10 monclient: get_auth_request con 0x7f003406c870 auth_method 0 2023-12-01T17:04:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.616+0000 7f00477fe640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f003406c870 0x7f003406ed30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0048145580 tx=0x7f003c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:02.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.616+0000 7f0044ff9640 10 client.4251.objecter ms_handle_connect 0x7f003406c870 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 1 -- 172.21.15.67:0/4123559890 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0038057dd0 con 0x7f00480a8730 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 10 client.4251.objecter ms_dispatch 0x7f0048000b90 osd_map(21..21 src has 1..21) v4 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 3 client.4251.objecter handle_osd_map got epochs [21,21] > 0 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 3 client.4251.objecter handle_osd_map decoding full epoch 21 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 20 client.4251.objecter dump_active .. 0 homeless 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 1 -- 172.21.15.67:0/4123559890 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0038054110 con 0x7f00480a8730 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:02.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.630+0000 7f0044ff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:02.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.839+0000 7f004d79e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:04:02.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.839+0000 7f004d79e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:04:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.839+0000 7f004d79e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:02.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:02.839+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00481461a0 con 0x7f00480a8730 2023-12-01T17:04:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.021+0000 7f0044ff9640 1 -- 172.21.15.67:0/4123559890 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14590 (secure 0 0 0) 0x7f0038021090 con 0x7f00480a8730 2023-12-01T17:04:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.021+0000 7f0044ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:04:03.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.021+0000 7f0044ff9640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:03.023 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:04:03.023 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:04:01.860525+0000","last_up_change":"2023-12-01T17:03:52.677540+0000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-01T17:03:50.595091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T17:03:58.823298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6811","nonce":358761697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6813","nonce":358761697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6817","nonce":358761697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6815","nonce":358761697}]},"public_addr":"172.21.15.67:6811/358761697","cluster_addr":"172.21.15.67:6813/358761697","heartbeat_back_addr":"172.21.15.67:6817/358761697","heartbeat_front_addr":"172.21.15.67:6815/358761697","state":["exists","up"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6827","nonce":866081726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6829","nonce":866081726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6833","nonce":866081726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6831","nonce":866081726}]},"public_addr":"172.21.15.67:6827/866081726","cluster_addr":"172.21.15.67:6829/866081726","heartbeat_back_addr":"172.21.15.67:6833/866081726","heartbeat_front_addr":"172.21.15.67:6831/866081726","state":["exists","up"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6819","nonce":2342127620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6821","nonce":2342127620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6825","nonce":2342127620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6823","nonce":2342127620}]},"public_addr":"172.21.15.67:6819/2342127620","cluster_addr":"172.21.15.67:6821/2342127620","heartbeat_back_addr":"172.21.15.67:6825/2342127620","heartbeat_front_addr":"172.21.15.67:6823/2342127620","state":["exists","up"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6803","nonce":1625009733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6805","nonce":1625009733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6809","nonce":1625009733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6807","nonce":1625009733}]},"public_addr":"172.21.15.67:6803/1625009733","cluster_addr":"172.21.15.67:6805/1625009733","heartbeat_back_addr":"172.21.15.67:6809/1625009733","heartbeat_front_addr":"172.21.15.67:6807/1625009733","state":["exists","up"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6817","nonce":2070368587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6819","nonce":2070368587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6823","nonce":2070368587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6821","nonce":2070368587}]},"public_addr":"172.21.15.92:6817/2070368587","cluster_addr":"172.21.15.92:6819/2070368587","heartbeat_back_addr":"172.21.15.92:6823/2070368587","heartbeat_front_addr":"172.21.15.92:6821/2070368587","state":["exists","up"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6801","nonce":1636405563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6803","nonce":1636405563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6807","nonce":1636405563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6805","nonce":1636405563}]},"public_addr":"172.21.15.92:6801/1636405563","cluster_addr":"172.21.15.92:6803/1636405563","heartbeat_back_addr":"172.21.15.92:6807/1636405563","heartbeat_front_addr":"172.21.15.92:6805/1636405563","state":["exists","up"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6825","nonce":162801920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6827","nonce":162801920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6831","nonce":162801920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6829","nonce":162801920}]},"public_addr":"172.21.15.92:6825/162801920","cluster_addr":"172.21.15.92:6827/162801920","heartbeat_back_addr":"172.21.15.92:6831/162801920","heartbeat_front_addr":"172.21.15.92:6829/162801920","state":["exists","up"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6809","nonce":2695530702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6811","nonce":2695530702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6815","nonce":2695530702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6813","nonce":2695530702}]},"public_addr":"172.21.15.92:6809/2695530702","cluster_addr":"172.21.15.92:6811/2695530702","heartbeat_back_addr":"172.21.15.92:6815/2695530702","heartbeat_front_addr":"172.21.15.92:6813/2695530702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.603058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.949751+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.632542+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.910362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.308141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.751004+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.105081+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.199584+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:04:03.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 10 librados: watch_flush enter 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 10 librados: watch_flush exit 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 20 client.4251.objecter shutdown clearing up homeless session... 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 10 client.4251.objecter successfully canceled tick 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f003406c870 msgr2=0x7f003406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f003406c870 0x7f003406ed30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0048145580 tx=0x7f003c009040 comp rx=0 tx=0).stop 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 10 monclient: shutdown 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00480a8730 msgr2=0x7f00480ab810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00480a8730 0x7f00480ab810 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f003800c440 tx=0x7f0038001e70 comp rx=0 tx=0).stop 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 shutdown_connections 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f004813f410 0x7f0048143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f003406c870 0x7f003406ed30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f004813efa0 0x7f00480abdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 --2- 172.21.15.67:0/4123559890 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f00480a8730 0x7f00480ab810 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 >> 172.21.15.67:0/4123559890 conn(0x7f00480b77a0 msgr2=0x7f004813bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 shutdown_connections 2023-12-01T17:04:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 -- 172.21.15.67:0/4123559890 wait complete. 2023-12-01T17:04:03.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.024+0000 7f004d79e640 1 librados: shutdown 2023-12-01T17:04:03.036 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-12-01T17:04:03.037 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T17:04:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.154+0000 7f7727b1d640 10 monclient: get_monmap_and_config 2023-12-01T17:04:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.154+0000 7f7727b1d640 10 monclient: build_initial_monmap 2023-12-01T17:04:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.154+0000 7f7727b1d640 10 monclient: monmap: 2023-12-01T17:04:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:03.155 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:03.156002+0000 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:03.156002+0000 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:03.156 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.154+0000 7f7727b1d640 1 Processor -- start 2023-12-01T17:04:03.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 1 -- start start 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient: init 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77200a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77201461c0 0x7f77200a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f77201461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f772013f0f0 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7720145620 con 0x7f77200a8a20 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200aace0 con 0x7f772013f0f0 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200aafc0 con 0x7f77201461c0 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.155+0000 7f7727b1d640 10 monclient(hunting): authenticate will time out at 1374.394128s 2023-12-01T17:04:03.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.156+0000 7f77250b8640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77201461c0 0x7f77200a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:03.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.156+0000 7f77260ba640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f772013f0f0 0x7f772013f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57734/0 (socket says 172.21.15.67:57734) 2023-12-01T17:04:03.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.156+0000 7f77260ba640 1 -- 172.21.15.67:0/2464424948 learned_addr learned my addr 172.21.15.67:0/2464424948 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:03.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.156+0000 7f77250b8640 10 monclient(hunting): get_auth_request con 0x7f77201461c0 auth_method 0 2023-12-01T17:04:03.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.156+0000 7f77250b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:03.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.156+0000 7f77250b8640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:03.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.156+0000 7f77250b8640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:03.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.157+0000 7f77250b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.157+0000 7f77250b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.157+0000 7f77250b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 1 -- 172.21.15.67:0/2464424948 >> v1:172.21.15.92:6789/0 conn(0x7f772013f0f0 legacy=0x7f772013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 1 -- 172.21.15.67:0/2464424948 >> v1:172.21.15.67:6789/0 conn(0x7f77200a8a20 legacy=0x7f7720145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 10 monclient: _finish_hunting 0 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 1 monclient: found mon.noname-b 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:03.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.158+0000 7f77250b8640 1 -- 172.21.15.67:0/2464424948 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77200b2210 con 0x7f77201461c0 2023-12-01T17:04:03.161 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.160+0000 7f77250b8640 1 --2- 172.21.15.67:0/2464424948 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77201461c0 0x7f77200a8e00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f77140097d0 tx=0x7f771402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f57f403594a7bdeb server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 1 -- 172.21.15.67:0/2464424948 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714031800 con 0x7f77201461c0 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: dump: 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:03.166 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: _finish_auth 0 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: _check_auth_tickets 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 1 -- 172.21.15.67:0/2464424948 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7714031960 con 0x7f77201461c0 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f7727b1d640 5 monclient: authenticate success, global_id 4244 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f7727b1d640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f7727b1d640 4 set_mon_vals no callback set 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 1 -- 172.21.15.67:0/2464424948 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7714031cf0 con 0x7f77201461c0 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.164+0000 7f77248b7640 10 monclient: dump: 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:03.167 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:03.168 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 10 monclient: shutdown 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 1 -- 172.21.15.67:0/2464424948 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77201461c0 msgr2=0x7f77200a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 1 --2- 172.21.15.67:0/2464424948 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77201461c0 0x7f77200a8e00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f77140097d0 tx=0x7f771402b8a0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 1 -- 172.21.15.67:0/2464424948 shutdown_connections 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 1 --2- 172.21.15.67:0/2464424948 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77201461c0 0x7f77200a8e00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 1 -- 172.21.15.67:0/2464424948 >> 172.21.15.67:0/2464424948 conn(0x7f77200b7890 msgr2=0x7f77200b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 1 -- 172.21.15.67:0/2464424948 shutdown_connections 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.165+0000 7f7727b1d640 1 -- 172.21.15.67:0/2464424948 wait complete. 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 10 monclient: build_initial_monmap 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 10 monclient: monmap: 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:03.171 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 1 librados: starting msgr at 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 1 librados: starting objecter 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 1 Processor -- start 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 1 -- start start 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 1 librados: setting wanted keys 2023-12-01T17:04:03.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 1 librados: calling monclient init 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.166+0000 7f7727b1d640 10 monclient: init 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77200a8a20 0x7f7720137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f77200a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f772013f0f0 0x7f772013fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f772013f0f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77201461c0 0x7f77201daaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f77201461c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7720145430 con 0x7f772013f0f0 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77201383a0 con 0x7f77201461c0 2023-12-01T17:04:03.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7720138eb0 con 0x7f77200a8a20 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.167+0000 7f7727b1d640 10 monclient(hunting): authenticate will time out at 1374.406042s 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77258b9640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77200a8a20 0x7f7720137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77260ba640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77201461c0 0x7f77201daaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77258b9640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77200a8a20 0x7f7720137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41712/0 (socket says 172.21.15.67:41712) 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77258b9640 1 -- 172.21.15.67:0/1223594857 learned_addr learned my addr 172.21.15.67:0/1223594857 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77258b9640 10 monclient(hunting): get_auth_request con 0x7f77200a8a20 auth_method 0 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77258b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77258b9640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77258b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77260ba640 10 monclient(hunting): get_auth_request con 0x7f77201461c0 auth_method 0 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77260ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:03.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77260ba640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.168+0000 7f77260ba640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.169+0000 7f77258b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.169+0000 7f77258b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.169+0000 7f77258b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.169+0000 7f77260ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.169+0000 7f77260ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.169+0000 7f77260ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:03.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.169+0000 7f77250b8640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f772013f0f0 0x7f772013fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.170+0000 7f77250b8640 10 monclient(hunting): get_auth_request con 0x7f772013f0f0 auth_method 0 2023-12-01T17:04:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.170+0000 7f77250b8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.170+0000 7f77250b8640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:03.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.170+0000 7f77250b8640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77250b8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77250b8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77250b8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 1 -- 172.21.15.67:0/1223594857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77201461c0 msgr2=0x7f77201daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77201461c0 0x7f77201daaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 1 -- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f772013f0f0 msgr2=0x7f772013fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f772013f0f0 0x7f772013fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 10 monclient: _finish_hunting 0 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 1 monclient: found mon.noname-c 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.172+0000 7f77258b9640 1 -- 172.21.15.67:0/1223594857 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7714009400 con 0x7f77200a8a20 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.174+0000 7f77258b9640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77200a8a20 0x7f7720137850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c410 tx=0x7f771c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:03.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.174+0000 7f77127fc640 10 client.?.objecter ms_handle_connect 0x7f77200a8a20 2023-12-01T17:04:03.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.174+0000 7f77127fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 1 -- 172.21.15.67:0/1223594857 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c0035f0 con 0x7f77200a8a20 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: dump: 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.181 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 1 -- 172.21.15.67:0/1223594857 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c003750 con 0x7f77200a8a20 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 5 monclient: authenticate success, global_id 4247 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 10 monclient: _renew_subs 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7720139380 con 0x7f77200a8a20 2023-12-01T17:04:03.182 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 1 -- 172.21.15.67:0/1223594857 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c003b40 con 0x7f77200a8a20 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f77127fc640 10 monclient: dump: 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:03.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7713fff640 4 set_mon_vals no callback set 2023-12-01T17:04:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 10 monclient: _renew_subs 2023-12-01T17:04:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77200b14d0 con 0x7f77200a8a20 2023-12-01T17:04:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.179+0000 7f7727b1d640 1 librados: init done 2023-12-01T17:04:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.180+0000 7f7727b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:03.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.180+0000 7f7727b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.180+0000 7f7727b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:03.185 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.180+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7720139180 con 0x7f77200a8a20 2023-12-01T17:04:03.200 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.199+0000 7f77127fc640 1 -- 172.21.15.67:0/1223594857 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f771c003ca0 con 0x7f77200a8a20 2023-12-01T17:04:03.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.200+0000 7f77127fc640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f770406c870 0x7f770406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:03.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.200+0000 7f77250b8640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f770406c870 0x7f770406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:03.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.200+0000 7f77250b8640 10 monclient: get_auth_request con 0x7f770406c870 auth_method 0 2023-12-01T17:04:03.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.200+0000 7f77250b8640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f770406c870 0x7f770406ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7714036000 tx=0x7f771403c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:03.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.200+0000 7f77127fc640 10 client.4247.objecter ms_handle_connect 0x7f770406c870 2023-12-01T17:04:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.202+0000 7f77127fc640 1 -- 172.21.15.67:0/1223594857 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f771c0010c0 con 0x7f77200a8a20 2023-12-01T17:04:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.202+0000 7f77127fc640 10 client.4247.objecter ms_dispatch 0x7f7720000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.202+0000 7f77127fc640 3 client.4247.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:03.203 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.202+0000 7f77127fc640 3 client.4247.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:03.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.202+0000 7f77127fc640 20 client.4247.objecter dump_active .. 0 homeless 2023-12-01T17:04:03.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.466+0000 7f77127fc640 1 -- 172.21.15.67:0/1223594857 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f771c054290 con 0x7f77200a8a20 2023-12-01T17:04:03.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.466+0000 7f77127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:03.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.466+0000 7f77127fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.649+0000 7f7727b1d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:04:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.649+0000 7f7727b1d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:04:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.649+0000 7f7727b1d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:03.651 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.649+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f772004e840 con 0x7f77200a8a20 2023-12-01T17:04:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.749+0000 7f77127fc640 1 -- 172.21.15.67:0/1223594857 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+14590 (secure 0 0 0) 0x7f771c017090 con 0x7f77200a8a20 2023-12-01T17:04:03.750 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.749+0000 7f77127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:04:03.751 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.749+0000 7f77127fc640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:03.751 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:04:03.751 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:04:01.860525+0000","last_up_change":"2023-12-01T17:03:52.677540+0000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-01T17:03:50.595091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T17:03:58.823298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6811","nonce":358761697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6813","nonce":358761697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6817","nonce":358761697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6815","nonce":358761697}]},"public_addr":"172.21.15.67:6811/358761697","cluster_addr":"172.21.15.67:6813/358761697","heartbeat_back_addr":"172.21.15.67:6817/358761697","heartbeat_front_addr":"172.21.15.67:6815/358761697","state":["exists","up"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6827","nonce":866081726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6829","nonce":866081726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6833","nonce":866081726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6831","nonce":866081726}]},"public_addr":"172.21.15.67:6827/866081726","cluster_addr":"172.21.15.67:6829/866081726","heartbeat_back_addr":"172.21.15.67:6833/866081726","heartbeat_front_addr":"172.21.15.67:6831/866081726","state":["exists","up"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6819","nonce":2342127620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6821","nonce":2342127620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6825","nonce":2342127620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6823","nonce":2342127620}]},"public_addr":"172.21.15.67:6819/2342127620","cluster_addr":"172.21.15.67:6821/2342127620","heartbeat_back_addr":"172.21.15.67:6825/2342127620","heartbeat_front_addr":"172.21.15.67:6823/2342127620","state":["exists","up"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6803","nonce":1625009733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6805","nonce":1625009733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6809","nonce":1625009733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6807","nonce":1625009733}]},"public_addr":"172.21.15.67:6803/1625009733","cluster_addr":"172.21.15.67:6805/1625009733","heartbeat_back_addr":"172.21.15.67:6809/1625009733","heartbeat_front_addr":"172.21.15.67:6807/1625009733","state":["exists","up"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6817","nonce":2070368587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6819","nonce":2070368587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6823","nonce":2070368587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6821","nonce":2070368587}]},"public_addr":"172.21.15.92:6817/2070368587","cluster_addr":"172.21.15.92:6819/2070368587","heartbeat_back_addr":"172.21.15.92:6823/2070368587","heartbeat_front_addr":"172.21.15.92:6821/2070368587","state":["exists","up"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6801","nonce":1636405563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6803","nonce":1636405563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6807","nonce":1636405563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6805","nonce":1636405563}]},"public_addr":"172.21.15.92:6801/1636405563","cluster_addr":"172.21.15.92:6803/1636405563","heartbeat_back_addr":"172.21.15.92:6807/1636405563","heartbeat_front_addr":"172.21.15.92:6805/1636405563","state":["exists","up"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6825","nonce":162801920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6827","nonce":162801920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6831","nonce":162801920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6829","nonce":162801920}]},"public_addr":"172.21.15.92:6825/162801920","cluster_addr":"172.21.15.92:6827/162801920","heartbeat_back_addr":"172.21.15.92:6831/162801920","heartbeat_front_addr":"172.21.15.92:6829/162801920","state":["exists","up"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6809","nonce":2695530702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6811","nonce":2695530702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6815","nonce":2695530702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6813","nonce":2695530702}]},"public_addr":"172.21.15.92:6809/2695530702","cluster_addr":"172.21.15.92:6811/2695530702","heartbeat_back_addr":"172.21.15.92:6815/2695530702","heartbeat_front_addr":"172.21.15.92:6813/2695530702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.603058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.949751+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.632542+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.910362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.308141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.751004+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.105081+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.199584+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 10 librados: watch_flush enter 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 10 librados: watch_flush exit 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 20 client.4247.objecter shutdown clearing up homeless session... 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 10 client.4247.objecter successfully canceled tick 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f770406c870 msgr2=0x7f770406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f770406c870 0x7f770406ed30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7714036000 tx=0x7f771403c040 comp rx=0 tx=0).stop 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 10 monclient: shutdown 2023-12-01T17:04:03.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77200a8a20 msgr2=0x7f7720137850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77200a8a20 0x7f7720137850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f771c00c410 tx=0x7f771c00c8d0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 shutdown_connections 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f77201461c0 0x7f77201daaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f772013f0f0 0x7f772013fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f770406c870 0x7f770406ed30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 --2- 172.21.15.67:0/1223594857 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f77200a8a20 0x7f7720137850 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 >> 172.21.15.67:0/1223594857 conn(0x7f77200b7890 msgr2=0x7f77200b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 shutdown_connections 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 -- 172.21.15.67:0/1223594857 wait complete. 2023-12-01T17:04:03.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.751+0000 7f7727b1d640 1 librados: shutdown 2023-12-01T17:04:03.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-12-01T17:04:03.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-12-01T17:04:03.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-12-01T17:04:03.766 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-12-01T17:04:03.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-12-01T17:04:03.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-12-01T17:04:03.767 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-12-01T17:04:03.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.994+0000 7fbbf5589640 10 monclient: get_monmap_and_config 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.994+0000 7fbbf5589640 10 monclient: build_initial_monmap 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.994+0000 7fbbf5589640 10 monclient: monmap: 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:03.996441+0000 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:03.996441+0000 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:03.996 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:03.997 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:03.997 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:03.997 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:03.997 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:03.997 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.996+0000 7fbbf5589640 1 Processor -- start 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.996+0000 7fbbf5589640 1 -- start start 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.996+0000 7fbbf5589640 10 monclient: init 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 0x7fbbf0143710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf00b0f40 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbf01468a0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:03.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbbf00b2920 0x7fbbf0143c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf00b2920 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf006ebc0 con 0x7fbbf01468a0 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf00b1320 con 0x7fbbf00b2920 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.997+0000 7fbbf5589640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf006eea0 con 0x7fbbf00b0f40 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.998+0000 7fbbf5589640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.998+0000 7fbbf5589640 10 monclient(hunting): authenticate will time out at 1375.236422s 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.998+0000 7fd6817b3640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.998+0000 7fd6817b3640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.998+0000 7fd6817b3640 10 monclient: monmap: 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.000469+0000 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.000469+0000 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.001 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.999+0000 7fbbef7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbbf00b2920 0x7fbbf0143c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.999+0000 7fbbef7fe640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbbf00b2920 0x7fbbf0143c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36476/0 (socket says 172.21.15.67:36476) 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.999+0000 7fbbef7fe640 1 -- 172.21.15.67:0/4150878299 learned_addr learned my addr 172.21.15.67:0/4150878299 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.999+0000 7fbbef7fe640 10 monclient(hunting): get_auth_request con 0x7fbbf00b2920 auth_method 0 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.999+0000 7fbbef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.999+0000 7fbbef7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:03.999+0000 7fbbef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.000+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.000+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.000+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.000+0000 7f846094e640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.000+0000 7f846094e640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.000+0000 7f846094e640 10 monclient: monmap: 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.001855+0000 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.001855+0000 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.003 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7fbbeeffd640 1 --2- 172.21.15.67:0/4150878299 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 0x7fbbf0143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7fbbeeffd640 10 monclient(hunting): get_auth_request con 0x7fbbf00b0f40 auth_method 0 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7fbbeeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7fbbeeffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7fbbeeffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7f846094e640 1 Processor -- start 2023-12-01T17:04:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7f846094e640 1 -- start start 2023-12-01T17:04:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.001+0000 7f846094e640 10 monclient: init 2023-12-01T17:04:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-12-01T17:04:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f845c0acb40 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f845c145c50 0x7f845c13b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.006 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient(hunting): picked mon.noname-a con 0x7f845c145c50 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f845c13ba20 0x7f845c13be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient(hunting): picked mon.noname-c con 0x7f845c13ba20 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c072b20 con 0x7f845c0acb40 2023-12-01T17:04:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c06fda0 con 0x7f845c145c50 2023-12-01T17:04:04.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f845c0acf20 con 0x7f845c13ba20 2023-12-01T17:04:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.002+0000 7f846094e640 10 monclient(hunting): authenticate will time out at 1375.241152s 2023-12-01T17:04:04.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-12-01T17:04:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.004+0000 7f845ad9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f845c13ba20 0x7f845c13be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.004+0000 7f845ad9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f845c13ba20 0x7f845c13be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36482/0 (socket says 172.21.15.67:36482) 2023-12-01T17:04:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.004+0000 7f845ad9d640 1 -- 172.21.15.67:0/1329127497 learned_addr learned my addr 172.21.15.67:0/1329127497 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.004+0000 7f845ad9d640 10 monclient(hunting): get_auth_request con 0x7f845c13ba20 auth_method 0 2023-12-01T17:04:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.004+0000 7f845ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.004+0000 7f845ad9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.005+0000 7f845ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 1 -- 172.21.15.67:0/4150878299 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 msgr2=0x7fbbf0143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 1 --2- 172.21.15.67:0/4150878299 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 0x7fbbf0143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 1 -- 172.21.15.67:0/4150878299 >> v1:172.21.15.67:6789/0 conn(0x7fbbf01468a0 legacy=0x7fbbf00b2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.003+0000 7fbbef7fe640 1 -- 172.21.15.67:0/4150878299 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf0146d80 con 0x7fbbf00b2920 2023-12-01T17:04:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.005+0000 7fd6817b3640 1 Processor -- start 2023-12-01T17:04:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.005+0000 7fd6817b3640 1 -- start start 2023-12-01T17:04:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.005+0000 7fd6817b3640 10 monclient: init 2023-12-01T17:04:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-12-01T17:04:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd67c06a770 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c146d70 0x7fd67c149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient(hunting): picked mon.noname-b con 0x7fd67c146d70 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd67c14a020 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67c149650 con 0x7fd67c06a770 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67c0702d0 con 0x7fd67c14a020 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c143df0 con 0x7fd67c146d70 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.006+0000 7fd6817b3640 10 monclient(hunting): authenticate will time out at 1375.245278s 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.007+0000 7f845ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.007+0000 7f845ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.007+0000 7f845ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.007+0000 7fbbef7fe640 1 --2- 172.21.15.67:0/4150878299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf0143c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009e80 tx=0x7fbbe802be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b99cb0d8d5ef524 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.007+0000 7fd680fb2640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd67c14a020 0x7fd67c150980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57736/0 (socket says 172.21.15.67:57736) 2023-12-01T17:04:04.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.007+0000 7fd680fb2640 1 -- 172.21.15.67:0/835960555 learned_addr learned my addr 172.21.15.67:0/835960555 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 10 monclient(hunting): handle_auth_done global_id 4306 payload 274 2023-12-01T17:04:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 1 -- 172.21.15.67:0/1329127497 >> v2:172.21.15.67:3300/0 conn(0x7f845c145c50 msgr2=0x7f845c13b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 1 --2- 172.21.15.67:0/1329127497 >> v2:172.21.15.67:3300/0 conn(0x7f845c145c50 0x7f845c13b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 1 -- 172.21.15.67:0/1329127497 >> v1:172.21.15.67:6789/0 conn(0x7f845c0acb40 legacy=0x7f845c145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7f845ad9d640 1 -- 172.21.15.67:0/1329127497 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c06e5c0 con 0x7f845c13ba20 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7fe76dcc2640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7fe76dcc2640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.009+0000 7fe76dcc2640 10 monclient: monmap: 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.011142+0000 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.011142+0000 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.010+0000 7fe76dcc2640 1 Processor -- start 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.010+0000 7fe76dcc2640 1 -- start start 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.010+0000 7fe76dcc2640 10 monclient: init 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fe7680af630 0x7fe76806fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient(hunting): picked mon.noname-a con 0x7fe7680af630 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe768071830 0x7fe768152440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient(hunting): picked mon.noname-b con 0x7fe768071830 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe768152980 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe76806ed00 con 0x7fe768152980 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe768070e50 con 0x7fe7680af630 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe768071130 con 0x7fe768071830 2023-12-01T17:04:04.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.011+0000 7fe76dcc2640 10 monclient(hunting): authenticate will time out at 1375.250117s 2023-12-01T17:04:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fe766ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe768071830 0x7fe768152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fe766ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe768071830 0x7fe768152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41738/0 (socket says 172.21.15.67:41738) 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fe766ffd640 1 -- 172.21.15.67:0/466159458 learned_addr learned my addr 172.21.15.67:0/466159458 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fe766ffd640 10 monclient(hunting): get_auth_request con 0x7fe768071830 auth_method 0 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fe766ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fe766ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fe766ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fd67b7fe640 1 --2- 172.21.15.67:0/835960555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c146d70 0x7fd67c149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fd67b7fe640 10 monclient(hunting): get_auth_request con 0x7fd67c146d70 auth_method 0 2023-12-01T17:04:04.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fd67b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fd67b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fd67b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 1 -- 172.21.15.67:0/4150878299 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe802c9e0 con 0x7fbbf00b2920 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: dump: 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.024 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 1 -- 172.21.15.67:0/4150878299 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe802cb40 con 0x7fbbf00b2920 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 1 -- 172.21.15.67:0/4150878299 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe80343f0 con 0x7fbbf00b2920 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbedffb640 10 monclient: dump: 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.025 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbf5589640 5 monclient: authenticate success, global_id 4303 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbf5589640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.017+0000 7fbbf5589640 4 set_mon_vals no callback set 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 10 monclient: shutdown 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 -- 172.21.15.67:0/4150878299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 msgr2=0x7fbbf0143c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 --2- 172.21.15.67:0/4150878299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf0143c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbbe8009e80 tx=0x7fbbe802be40 comp rx=0 tx=0).stop 2023-12-01T17:04:04.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 -- 172.21.15.67:0/4150878299 shutdown_connections 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 --2- 172.21.15.67:0/4150878299 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf0143c50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 --2- 172.21.15.67:0/4150878299 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 0x7fbbf0143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 -- 172.21.15.67:0/4150878299 >> 172.21.15.67:0/4150878299 conn(0x7fbbf00b7890 msgr2=0x7fbbf00b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 -- 172.21.15.67:0/4150878299 shutdown_connections 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.018+0000 7fbbf5589640 1 -- 172.21.15.67:0/4150878299 wait complete. 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.019+0000 7fbbf5589640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.019+0000 7fbbf5589640 10 monclient: monmap: 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.027 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.019+0000 7fbbf5589640 1 librados: starting msgr at 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.019+0000 7fbbf5589640 1 librados: starting objecter 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fbbf5589640 1 Processor -- start 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fbbf5589640 1 -- start start 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fbbf5589640 1 librados: setting wanted keys 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fbbf5589640 1 librados: calling monclient init 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.020+0000 7fbbf5589640 10 monclient: init 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 0x7fbbf01f56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): picked mon.noname-c con 0x7fbbf00b0f40 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf00abf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): picked mon.noname-b con 0x7fbbf00b2920 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbf01468a0 0x7fbbf00ac4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): picked mon.noname-a con 0x7fbbf01468a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00ac9f0 con 0x7fbbf01468a0 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00adc40 con 0x7fbbf00b2920 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf00ab500 con 0x7fbbf00b0f40 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbf5589640 10 monclient(hunting): authenticate will time out at 1375.259505s 2023-12-01T17:04:04.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7f845ad9d640 1 --2- 172.21.15.67:0/1329127497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 0x7f845c13be90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f845c1460b0 tx=0x7f844c02b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c433038ed1b76e5 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.021+0000 7fbbee7fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf00abf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.022+0000 7fbbee7fc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf00abf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36484/0 (socket says 172.21.15.67:36484) 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.022+0000 7fbbee7fc640 1 -- 172.21.15.67:0/4145107819 learned_addr learned my addr 172.21.15.67:0/4145107819 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.022+0000 7fbbee7fc640 10 monclient(hunting): get_auth_request con 0x7fbbf00b2920 auth_method 0 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.022+0000 7fbbee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.022+0000 7fbbee7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.022+0000 7fbbee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.023+0000 7fbbee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.023+0000 7fbbee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.023+0000 7fbbee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.023+0000 7fd67b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.023+0000 7fd67b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.023+0000 7fd67b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.024+0000 7fe766ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.024+0000 7fe766ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.024+0000 7fe766ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2023-12-01T17:04:04.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 1 -- 172.21.15.67:0/466159458 >> v2:172.21.15.67:3300/0 conn(0x7fe7680af630 msgr2=0x7fe76806fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 1 --2- 172.21.15.67:0/466159458 >> v2:172.21.15.67:3300/0 conn(0x7fe7680af630 0x7fe76806fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 1 -- 172.21.15.67:0/466159458 >> v1:172.21.15.67:6789/0 conn(0x7fe768152980 legacy=0x7fe7680ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.026+0000 7fe766ffd640 1 -- 172.21.15.67:0/466159458 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768071410 con 0x7fe768071830 2023-12-01T17:04:04.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 10 monclient(hunting): handle_auth_done global_id 4309 payload 824 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 1 -- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 msgr2=0x7fbbf01f56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbbf00b0f40 0x7fbbf01f56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 1 -- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbf01468a0 msgr2=0x7fbbf00ac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbf01468a0 0x7fbbf00ac4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7fbbee7fc640 1 -- 172.21.15.67:0/4145107819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbe8009b30 con 0x7fbbf00b2920 2023-12-01T17:04:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7f11dffff640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7f11dffff640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.028+0000 7f11dffff640 10 monclient: monmap: 2023-12-01T17:04:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.030401+0000 2023-12-01T17:04:04.035 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.030401+0000 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.029+0000 7f11dffff640 1 Processor -- start 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.029+0000 7f11dffff640 1 -- start start 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.029+0000 7f11dffff640 10 monclient: init 2023-12-01T17:04:04.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f845959a640 1 -- 172.21.15.67:0/1329127497 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c031870 con 0x7f845c13ba20 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f845959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f845959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f845959a640 10 monclient: dump: 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f845959a640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.037 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f845959a640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.038 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f845959a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.038 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f846094e640 5 monclient: authenticate success, global_id 4306 2023-12-01T17:04:04.038 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f846094e640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.038 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7fbbee7fc640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf00abf70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400b3e0 tx=0x7fbbe400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.038 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7fbbd77fe640 10 client.?.objecter ms_handle_connect 0x7fbbf00b2920 2023-12-01T17:04:04.038 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7fbbd77fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f11dffff640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f767a02b640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f767a02b640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.030+0000 7f767a02b640 10 monclient: monmap: 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.032528+0000 2023-12-01T17:04:04.039 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.032528+0000 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.040 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f11d8038da0 0x7f11d80c5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f11d8038da0 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11d8039180 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f11d80c5790 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80c7e90 con 0x7f11d80c5790 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8035890 con 0x7f11d8039180 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d8037630 con 0x7f11d8038da0 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f11dffff640 10 monclient(hunting): authenticate will time out at 1375.269872s 2023-12-01T17:04:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f767a02b640 1 Processor -- start 2023-12-01T17:04:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f767a02b640 1 -- start start 2023-12-01T17:04:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.031+0000 7f767a02b640 10 monclient: init 2023-12-01T17:04:04.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7f04fadd2640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7f04fadd2640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7f04fadd2640 10 monclient: monmap: 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.034285+0000 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.034285+0000 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.044 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.045 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.045 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7fd67affd640 1 -- 172.21.15.67:0/835960555 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918730127 0 0) 0x7fd67c0702d0 con 0x7fd67c14a020 2023-12-01T17:04:04.045 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7fd67affd640 10 monclient(hunting): _negotiate 2023-12-01T17:04:04.045 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7fd67affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.045 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7fd67affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.045 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7fd67affd640 10 monclient(hunting): my global_id is 4312 2023-12-01T17:04:04.046 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7fd67affd640 1 -- 172.21.15.67:0/835960555 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd664003520 con 0x7fd67c14a020 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.032+0000 7f767a02b640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f76740af380 0x7f767406fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): picked mon.noname-a con 0x7f76740af380 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7674072090 0x7f76740724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): picked mon.noname-b con 0x7f7674072090 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f767413b4c0 0x7f767413b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): picked mon.noname-c con 0x7f767413b4c0 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:04.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f04fadd2640 1 Processor -- start 2023-12-01T17:04:04.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76740af760 con 0x7f76740af380 2023-12-01T17:04:04.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76740702b0 con 0x7f767413b4c0 2023-12-01T17:04:04.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7674070590 con 0x7f7674072090 2023-12-01T17:04:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.033+0000 7f767a02b640 10 monclient(hunting): authenticate will time out at 1375.272121s 2023-12-01T17:04:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f04fadd2640 1 -- start start 2023-12-01T17:04:04.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f04fadd2640 10 monclient: init 2023-12-01T17:04:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f845959a640 1 -- 172.21.15.67:0/1329127497 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844c0319d0 con 0x7f845c13ba20 2023-12-01T17:04:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f845959a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f845959a640 1 -- 172.21.15.67:0/1329127497 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c031d60 con 0x7f845c13ba20 2023-12-01T17:04:04.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f845959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f845959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f845959a640 10 monclient: dump: 2023-12-01T17:04:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.052 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.053 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f846094e640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.053 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f846094e640 4 set_mon_vals no callback set 2023-12-01T17:04:04.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.034+0000 7f04fadd2640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-12-01T17:04:04.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f04ec0c5560 0x7f04ec007210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec0c5560 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec0c7fd0 0x7f04ec0c8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient(hunting): picked mon.noname-b con 0x7f04ec0c7fd0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f04ec036bf0 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:04.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04ec0cb130 con 0x7f04ec036bf0 2023-12-01T17:04:04.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04ec0cc120 con 0x7f04ec0c5560 2023-12-01T17:04:04.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec009370 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f04fadd2640 10 monclient(hunting): authenticate will time out at 1375.273518s 2023-12-01T17:04:04.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 10 monclient: shutdown 2023-12-01T17:04:04.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 -- 172.21.15.67:0/1329127497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 msgr2=0x7f845c13be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 --2- 172.21.15.67:0/1329127497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 0x7f845c13be90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f845c1460b0 tx=0x7f844c02b910 comp rx=0 tx=0).stop 2023-12-01T17:04:04.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 -- 172.21.15.67:0/1329127497 shutdown_connections 2023-12-01T17:04:04.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 --2- 172.21.15.67:0/1329127497 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 0x7f845c13be90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 --2- 172.21.15.67:0/1329127497 >> v2:172.21.15.67:3300/0 conn(0x7f845c145c50 0x7f845c13b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 -- 172.21.15.67:0/1329127497 >> 172.21.15.67:0/1329127497 conn(0x7f845c0b3b20 msgr2=0x7f845c0b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 -- 172.21.15.67:0/1329127497 shutdown_connections 2023-12-01T17:04:04.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7f846094e640 1 -- 172.21.15.67:0/1329127497 wait complete. 2023-12-01T17:04:04.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7fd67b7fe640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2023-12-01T17:04:04.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7fd67b7fe640 1 -- 172.21.15.67:0/835960555 >> v1:172.21.15.92:6789/0 conn(0x7fd67c14a020 legacy=0x7fd67c150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7fd67b7fe640 1 -- 172.21.15.67:0/835960555 >> v1:172.21.15.67:6789/0 conn(0x7fd67c06a770 legacy=0x7fd67c146660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7fd67b7fe640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7fd67b7fe640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7fd67b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.035+0000 7fd67b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.036+0000 7fd67b7fe640 1 -- 172.21.15.67:0/835960555 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67c0705b0 con 0x7fd67c146d70 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.036+0000 7f846094e640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.036+0000 7f846094e640 10 monclient: monmap: 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.061 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 librados: starting msgr at 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 librados: starting objecter 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 Processor -- start 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 -- start start 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 librados: setting wanted keys 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 librados: calling monclient init 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient: init 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c0acb40 0x7f845c1416d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): picked mon.noname-a con 0x7f845c0acb40 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 0x7f845c140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): picked mon.noname-b con 0x7f845c13ba20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f845c145c50 0x7f845c1d7730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): picked mon.noname-c con 0x7f845c145c50 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1429c0 con 0x7f845c0acb40 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f845c1d7210 con 0x7f845c13ba20 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f845c13fdc0 con 0x7f845c145c50 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.037+0000 7f846094e640 10 monclient(hunting): authenticate will time out at 1375.276325s 2023-12-01T17:04:04.064 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.040+0000 7f11de7fc640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f11d8039180 0x7f11d80c8340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57740/0 (socket says 172.21.15.67:57740) 2023-12-01T17:04:04.064 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.040+0000 7f11de7fc640 1 -- 172.21.15.67:0/3385049357 learned_addr learned my addr 172.21.15.67:0/3385049357 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:04.064 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.041+0000 7f8459d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 0x7f845c140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.064 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.042+0000 7fd67b7fe640 1 --2- 172.21.15.67:0/835960555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c146d70 0x7fd67c149ae0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0097d0 tx=0x7fd66c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77b68c0c04706282 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.065 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.042+0000 7f845ad9d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f845c145c50 0x7f845c1d7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.065 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.042+0000 7f7672ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7674072090 0x7f76740724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.065 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.042+0000 7f04fa5d1640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f04ec036bf0 0x7f04ec037080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57756/0 (socket says 172.21.15.67:57756) 2023-12-01T17:04:04.065 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.042+0000 7f04fa5d1640 1 -- 172.21.15.67:0/3782567203 learned_addr learned my addr 172.21.15.67:0/3782567203 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:04.066 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f845ad9d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f845c145c50 0x7f845c1d7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41786/0 (socket says 172.21.15.67:41786) 2023-12-01T17:04:04.066 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f845ad9d640 1 -- 172.21.15.67:0/2603768082 learned_addr learned my addr 172.21.15.67:0/2603768082 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.066 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f845ad9d640 10 monclient(hunting): get_auth_request con 0x7f845c145c50 auth_method 0 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f845ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f845ad9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f845ad9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f7672ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7674072090 0x7f76740724b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41762/0 (socket says 172.21.15.67:41762) 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f7672ffd640 1 -- 172.21.15.67:0/991810357 learned_addr learned my addr 172.21.15.67:0/991810357 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f7672ffd640 10 monclient(hunting): get_auth_request con 0x7f7674072090 auth_method 0 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f7672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f7672ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.067 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.043+0000 7f7672ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.068 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f7672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.068 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f7672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.068 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f7672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4011830 con 0x7fbbf00b2920 2023-12-01T17:04:04.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: dump: 2023-12-01T17:04:04.069 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.069 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.069 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe40119b0 con 0x7fbbf00b2920 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 5 monclient: authenticate success, global_id 4309 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 10 monclient: _renew_subs 2023-12-01T17:04:04.070 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf01f51d0 con 0x7fbbf00b2920 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4011d60 con 0x7fbbf00b2920 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbd77fe640 10 monclient: dump: 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.071 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 10 monclient: _renew_subs 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf00a8e90 con 0x7fbbf00b2920 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbf5589640 1 librados: init done 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7fbbecff9640 4 set_mon_vals no callback set 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f7673fff640 1 --2- 172.21.15.67:0/991810357 >> v2:172.21.15.92:3300/0 conn(0x7f767413b4c0 0x7f767413b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f8459d9b640 10 monclient(hunting): get_auth_request con 0x7f845c13ba20 auth_method 0 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f8459d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f8459d9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.044+0000 7f8459d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.045+0000 7f7673fff640 10 monclient(hunting): get_auth_request con 0x7f767413b4c0 auth_method 0 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.045+0000 7f7673fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.045+0000 7f7673fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.045+0000 7f7673fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.045+0000 7f8459d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.045+0000 7f8459d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f8459d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7fbbf5589640 10 client.4309.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7fbbf5589640 15 client.4309.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7fbbf5589640 20 client.4309.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7fbbf5589640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7fbbf5589640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbbf00b4070 con 0x7fbbf00b2920 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 1 -- 172.21.15.67:0/991810357 >> v2:172.21.15.92:3300/0 conn(0x7f767413b4c0 msgr2=0x7f767413b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 1 --2- 172.21.15.67:0/991810357 >> v2:172.21.15.92:3300/0 conn(0x7f767413b4c0 0x7f767413b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 1 -- 172.21.15.67:0/991810357 >> v2:172.21.15.67:3300/0 conn(0x7f76740af380 msgr2=0x7f767406fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 1 --2- 172.21.15.67:0/991810357 >> v2:172.21.15.67:3300/0 conn(0x7f76740af380 0x7f767406fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.046+0000 7f7672ffd640 1 -- 172.21.15.67:0/991810357 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76740add10 con 0x7f7674072090 2023-12-01T17:04:04.076 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.047+0000 7f845ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.076 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.047+0000 7f845ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.076 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.047+0000 7f845ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.048+0000 7fea3d3ce640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.048+0000 7fea3d3ce640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.048+0000 7fea3d3ce640 10 monclient: monmap: 2023-12-01T17:04:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.077 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.050713+0000 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.050713+0000 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.078 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2023-12-01T17:04:04.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7fea3d3ce640 1 Processor -- start 2023-12-01T17:04:04.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 1 -- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f845c145c50 msgr2=0x7f845c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f845c145c50 0x7f845c1d7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 1 -- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c0acb40 msgr2=0x7f845c1416d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f845c0acb40 0x7f845c1416d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7f8459d9b640 1 -- 172.21.15.67:0/2603768082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f844c0094d0 con 0x7f845c13ba20 2023-12-01T17:04:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7fea3d3ce640 1 -- start start 2023-12-01T17:04:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.049+0000 7fea3d3ce640 10 monclient: init 2023-12-01T17:04:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-12-01T17:04:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fea38061190 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fea3806efa0 0x7fea38143410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient(hunting): picked mon.noname-a con 0x7fea3806efa0 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea3806f800 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea380705c0 con 0x7fea38061190 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea38070830 con 0x7fea3806f800 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea380717d0 con 0x7fea3806efa0 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7fea3d3ce640 10 monclient(hunting): authenticate will time out at 1375.289029s 2023-12-01T17:04:04.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.050+0000 7f04f95cf640 1 --2- 172.21.15.67:0/3782567203 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec0c7fd0 0x7f04ec0c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f7672ffd640 1 --2- 172.21.15.67:0/991810357 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7674072090 0x7f76740724b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f767406cde0 tx=0x7f766802cb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=261938706b2c8c66 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f04f95cf640 10 monclient(hunting): get_auth_request con 0x7f04ec0c7fd0 auth_method 0 2023-12-01T17:04:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f04f95cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f04f95cf640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f04f95cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f11ddffb640 1 -- 172.21.15.67:0/3385049357 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234205420 0 0) 0x7f11d8035890 con 0x7f11d8039180 2023-12-01T17:04:04.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f11ddffb640 10 monclient(hunting): _negotiate 2023-12-01T17:04:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f11ddffb640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f11ddffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f11ddffb640 10 monclient(hunting): my global_id is 4315 2023-12-01T17:04:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.051+0000 7f11ddffb640 1 -- 172.21.15.67:0/3385049357 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c8003520 con 0x7f11d8039180 2023-12-01T17:04:04.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.052+0000 7fe766ffd640 1 --2- 172.21.15.67:0/466159458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe768071830 0x7fe768152440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe760009e50 tx=0x7fe76002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f656267dca9d693d server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.052+0000 7f04f95cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.052+0000 7f04f95cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.052+0000 7f04f95cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.054+0000 7f04f8dce640 1 -- 172.21.15.67:0/3782567203 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076466643 0 0) 0x7f04ec0cb130 con 0x7f04ec036bf0 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.054+0000 7f04f8dce640 10 monclient(hunting): _negotiate 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.054+0000 7f04f8dce640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.054+0000 7f04f8dce640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.054+0000 7f04f8dce640 10 monclient(hunting): my global_id is 4321 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.054+0000 7f04f8dce640 1 -- 172.21.15.67:0/3782567203 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e0003520 con 0x7f04ec036bf0 2023-12-01T17:04:04.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.055+0000 7f8459d9b640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 0x7f845c140ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f845400bd70 tx=0x7f845400be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.055+0000 7f844affd640 10 client.?.objecter ms_handle_connect 0x7f845c13ba20 2023-12-01T17:04:04.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.055+0000 7f844affd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7fea36ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fea38061190 0x7fea3806e890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57512/0 (socket says 172.21.15.67:57512) 2023-12-01T17:04:04.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7fea36ffd640 1 -- 172.21.15.67:0/2032044017 learned_addr learned my addr 172.21.15.67:0/2032044017 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:04.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7f04f95cf640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2023-12-01T17:04:04.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7f04f95cf640 1 -- 172.21.15.67:0/3782567203 >> v2:172.21.15.67:3300/0 conn(0x7f04ec0c5560 msgr2=0x7f04ec007210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7f04f95cf640 1 --2- 172.21.15.67:0/3782567203 >> v2:172.21.15.67:3300/0 conn(0x7f04ec0c5560 0x7f04ec007210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7f04f95cf640 1 -- 172.21.15.67:0/3782567203 >> v1:172.21.15.92:6789/0 conn(0x7f04ec036bf0 legacy=0x7f04ec037080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7f04f95cf640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.056+0000 7f04f95cf640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.057+0000 7f04f95cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.057+0000 7f04f95cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.057+0000 7f04f95cf640 1 -- 172.21.15.67:0/3782567203 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec030680 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.057+0000 7f11deffd640 1 --2- 172.21.15.67:0/3385049357 >> v2:172.21.15.67:3300/0 conn(0x7f11d8038da0 0x7f11d80c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.058+0000 7f11deffd640 10 monclient(hunting): get_auth_request con 0x7f11d8038da0 auth_method 0 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.058+0000 7f11deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.058+0000 7f11deffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.058+0000 7f11deffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.062+0000 7f11ddffb640 1 -- 172.21.15.67:0/3385049357 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918134514 0 0) 0x7f11d80c7e90 con 0x7f11d80c5790 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.062+0000 7f11ddffb640 10 monclient(hunting): _negotiate 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.063+0000 7f11ddffb640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.063+0000 7f11ddffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.063+0000 7f11ddffb640 10 monclient(hunting): my global_id is 4260 2023-12-01T17:04:04.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.063+0000 7f11ddffb640 1 -- 172.21.15.67:0/3385049357 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d8035890 con 0x7f11d80c5790 2023-12-01T17:04:04.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.064+0000 7f04f95cf640 1 --2- 172.21.15.67:0/3782567203 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec0c7fd0 0x7f04ec0c8410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54bd8643fa7ef92d server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.064+0000 7fea2e7fc640 1 --2- 172.21.15.67:0/2032044017 >> v2:172.21.15.67:3300/0 conn(0x7fea3806efa0 0x7fea38143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.065+0000 7fea2e7fc640 10 monclient(hunting): get_auth_request con 0x7fea3806efa0 auth_method 0 2023-12-01T17:04:04.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.065+0000 7fea2e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.065+0000 7fea2e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.065+0000 7fea2e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.065+0000 7f11deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.065+0000 7f11deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.065+0000 7f11deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.068+0000 7fea2e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.068+0000 7fea2e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.068+0000 7fea2e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 10 monclient(hunting): handle_auth_done global_id 4263 payload 274 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 1 -- 172.21.15.67:0/3385049357 >> v1:172.21.15.92:6789/0 conn(0x7f11d8039180 legacy=0x7f11d80c8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 1 -- 172.21.15.67:0/3385049357 >> v1:172.21.15.67:6789/0 conn(0x7f11d80c5790 legacy=0x7f11d8034450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 1 monclient: found mon.noname-a 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.070+0000 7f11deffd640 1 -- 172.21.15.67:0/3385049357 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d8036090 con 0x7f11d8038da0 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.073+0000 7fea367fc640 1 -- 172.21.15.67:0/2032044017 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472459353 0 0) 0x7fea380705c0 con 0x7fea38061190 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.073+0000 7fea367fc640 10 monclient(hunting): _negotiate 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.073+0000 7fea367fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.073+0000 7fea367fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.073+0000 7fea367fc640 10 monclient(hunting): my global_id is 4266 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.073+0000 7fea367fc640 1 -- 172.21.15.67:0/2032044017 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea24003520 con 0x7fea38061190 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 1 -- 172.21.15.67:0/2032044017 >> v1:172.21.15.92:6789/0 conn(0x7fea3806f800 legacy=0x7fea38143950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 1 -- 172.21.15.67:0/2032044017 >> v1:172.21.15.67:6789/0 conn(0x7fea38061190 legacy=0x7fea3806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 1 monclient: found mon.noname-a 2023-12-01T17:04:04.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.074+0000 7fea2e7fc640 1 -- 172.21.15.67:0/2032044017 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38070b10 con 0x7fea3806efa0 2023-12-01T17:04:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.075+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbbe401a400 con 0x7fbbf00b2920 2023-12-01T17:04:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.076+0000 7fbbd77fe640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbbdc06c870 0x7fbbdc06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.076+0000 7fbbeeffd640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbbdc06c870 0x7fbbdc06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.076+0000 7fbbeeffd640 10 monclient: get_auth_request con 0x7fbbdc06c870 auth_method 0 2023-12-01T17:04:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.076+0000 7fbbeeffd640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbbdc06c870 0x7fbbdc06ed30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00098f0 tx=0x7fbbe0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.076+0000 7fbbd77fe640 10 client.4309.objecter ms_handle_connect 0x7fbbdc06c870 2023-12-01T17:04:04.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.079+0000 7f11deffd640 1 --2- 172.21.15.67:0/3385049357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 0x7f11d80c5250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11d00097d0 tx=0x7f11d002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1732982e187fe0be server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.080+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbbe4059140 con 0x7fbbf00b2920 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.080+0000 7fbbd77fe640 10 client.4309.objecter ms_dispatch 0x7fbbf0000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.080+0000 7fbbd77fe640 3 client.4309.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.080+0000 7fbbd77fe640 3 client.4309.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 10 client.4309.objecter checking command 1 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fbbdc072c50 0x7fbbdc075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 20 client.4309.objecter _get_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 20 client.4309.objecter put_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 15 client.4309.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 20 client.4309.objecter _get_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 20 client.4309.objecter get_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 15 client.4309.objecter _session_command_op_assign 5 1 2023-12-01T17:04:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 20 client.4309.objecter put_session s=0x7fbbdc072580 osd=5 4 2023-12-01T17:04:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 10 client.4309.objecter _send_command 1 2023-12-01T17:04:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 --> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbbdc075720 con 0x7fbbdc072c50 2023-12-01T17:04:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 20 client.4309.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.097 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fea2e7fc640 1 --2- 172.21.15.67:0/2032044017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806efa0 0x7fea38143410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea200097a0 tx=0x7fea2002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2802c041f92e056b server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbef7fe640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fbbdc072c50 0x7fbbdc075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbe4009210 con 0x7fbbf00b2920 2023-12-01T17:04:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbd77fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.081+0000 7fbbef7fe640 10 monclient: get_auth_request con 0x7fbbdc072c50 auth_method 0 2023-12-01T17:04:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.085+0000 7fbbd77fe640 10 client.4309.objecter ms_handle_connect 0x7fbbdc072c50 2023-12-01T17:04:04.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.085+0000 7fbbef7fe640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fbbdc072c50 0x7fbbdc075070 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845400ce90 con 0x7f845c13ba20 2023-12-01T17:04:04.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f844affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f844affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f844affd640 10 monclient: dump: 2023-12-01T17:04:04.099 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.099 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f844affd640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f844affd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f844affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 5 monclient: authenticate success, global_id 4318 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 10 monclient: _renew_subs 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c13d480 con 0x7f845c13ba20 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 10 monclient: _renew_subs 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c1d7c70 con 0x7f845c13ba20 2023-12-01T17:04:04.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.088+0000 7f846094e640 1 librados: init done 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11ddffb640 1 -- 172.21.15.67:0/3385049357 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d002d040 con 0x7f11d8038da0 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11ddffb640 10 monclient: dump: 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.102 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.103 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11ddffb640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11ddffb640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11dffff640 5 monclient: authenticate success, global_id 4263 2023-12-01T17:04:04.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f11dffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f846094e640 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f846094e640 15 client.4318.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f846094e640 20 client.4318.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f846094e640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f846094e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.089+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f845c0add20 con 0x7f845c13ba20 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.090+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8454009870 con 0x7f845c13ba20 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.090+0000 7f844affd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.090+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845401e640 con 0x7f845c13ba20 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.090+0000 7f844affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.090+0000 7f844affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.090+0000 7f844affd640 10 monclient: dump: 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.090+0000 7f8458d99640 4 set_mon_vals no callback set 2023-12-01T17:04:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11ddffb640 1 -- 172.21.15.67:0/3385049357 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d0031810 con 0x7f11d8038da0 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11ddffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11ddffb640 1 -- 172.21.15.67:0/3385049357 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d0031b80 con 0x7f11d8038da0 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11dffff640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11dffff640 4 set_mon_vals no callback set 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.093+0000 7f11ddffb640 10 monclient: dump: 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.106 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.107 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea367fc640 1 -- 172.21.15.67:0/2032044017 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2002f050 con 0x7fea3806efa0 2023-12-01T17:04:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea367fc640 10 monclient: dump: 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea367fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea367fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea3d3ce640 5 monclient: authenticate success, global_id 4269 2023-12-01T17:04:04.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7fea3d3ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 10 monclient: shutdown 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 1 -- 172.21.15.67:0/3385049357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 msgr2=0x7f11d80c5250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 1 --2- 172.21.15.67:0/3385049357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 0x7f11d80c5250 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f11d00097d0 tx=0x7f11d002b8b0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 1 -- 172.21.15.67:0/3385049357 shutdown_connections 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 1 --2- 172.21.15.67:0/3385049357 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 0x7f11d80c5250 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 1 -- 172.21.15.67:0/3385049357 >> 172.21.15.67:0/3385049357 conn(0x7f11d8006600 msgr2=0x7f11d8006a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 1 -- 172.21.15.67:0/3385049357 shutdown_connections 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.094+0000 7f11dffff640 1 -- 172.21.15.67:0/3385049357 wait complete. 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 10 monclient: monmap: 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.110 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 1 librados: starting msgr at 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 1 librados: starting objecter 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 1 Processor -- start 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 1 -- start start 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 1 librados: setting wanted keys 2023-12-01T17:04:04.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 1 librados: calling monclient init 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.095+0000 7f11dffff640 10 monclient: init 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea367fc640 1 -- 172.21.15.67:0/2032044017 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea20031700 con 0x7fea3806efa0 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea367fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea367fc640 1 -- 172.21.15.67:0/2032044017 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20031a70 con 0x7fea3806efa0 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea367fc640 10 monclient: dump: 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.112 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.113 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 0x7f11d80cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): picked mon.noname-a con 0x7f11d8038da0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11d80c5790 0x7f11d815d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): picked mon.noname-c con 0x7f11d80c5790 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d802d9d0 0x7f11d815d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): picked mon.noname-b con 0x7f11d802d9d0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d802ec50 con 0x7f11d8038da0 2023-12-01T17:04:04.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11d802ee50 con 0x7f11d802d9d0 2023-12-01T17:04:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d815cd20 con 0x7f11d80c5790 2023-12-01T17:04:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7f11dffff640 10 monclient(hunting): authenticate will time out at 1375.334876s 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 4 set_mon_vals no callback set 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 10 monclient: shutdown 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 1 -- 172.21.15.67:0/2032044017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806efa0 msgr2=0x7fea38143410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 1 --2- 172.21.15.67:0/2032044017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806efa0 0x7fea38143410 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fea200097a0 tx=0x7fea2002b7a0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 1 -- 172.21.15.67:0/2032044017 shutdown_connections 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 1 --2- 172.21.15.67:0/2032044017 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806efa0 0x7fea38143410 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.096+0000 7fea3d3ce640 1 -- 172.21.15.67:0/2032044017 >> 172.21.15.67:0/2032044017 conn(0x7fea3806a500 msgr2=0x7fea3806a910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7fea3d3ce640 1 -- 172.21.15.67:0/2032044017 shutdown_connections 2023-12-01T17:04:04.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7fea3d3ce640 1 -- 172.21.15.67:0/2032044017 wait complete. 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7f11de7fc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11d80c5790 0x7f11d815d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7fea3d3ce640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7fea3d3ce640 10 monclient: monmap: 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.117 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7f11de7fc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11d80c5790 0x7f11d815d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41802/0 (socket says 172.21.15.67:41802) 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7f11de7fc640 1 -- 172.21.15.67:0/3657935820 learned_addr learned my addr 172.21.15.67:0/3657935820 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7f11de7fc640 10 monclient(hunting): get_auth_request con 0x7f11d80c5790 auth_method 0 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7f11de7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7f11de7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.097+0000 7f11de7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7fea3d3ce640 1 librados: starting msgr at 2023-12-01T17:04:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7fea3d3ce640 1 librados: starting objecter 2023-12-01T17:04:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7fea3d3ce640 1 Processor -- start 2023-12-01T17:04:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7fea3d3ce640 1 -- start start 2023-12-01T17:04:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7fea3d3ce640 1 librados: setting wanted keys 2023-12-01T17:04:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7fea3d3ce640 1 librados: calling monclient init 2023-12-01T17:04:04.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7fea3d3ce640 10 monclient: init 2023-12-01T17:04:04.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7f11de7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7f11de7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.098+0000 7f11de7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea38061190 0x7fea38147860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): picked mon.noname-c con 0x7fea38061190 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea3806efa0 0x7fea38149650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): picked mon.noname-b con 0x7fea3806efa0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806f800 0x7fea38149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): picked mon.noname-a con 0x7fea3806f800 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea381489c0 con 0x7fea3806f800 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea38148ca0 con 0x7fea3806efa0 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea38148f80 con 0x7fea38061190 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.099+0000 7fea3d3ce640 10 monclient(hunting): authenticate will time out at 1375.337800s 2023-12-01T17:04:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.101+0000 7f11deffd640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 0x7f11d80cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.101+0000 7fea377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806f800 0x7fea38149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.101+0000 7f11deffd640 10 monclient(hunting): get_auth_request con 0x7f11d8038da0 auth_method 0 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.101+0000 7f11deffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.101+0000 7f11deffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.101+0000 7f11deffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11deffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11deffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11deffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 1 -- 172.21.15.67:0/3657935820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d802d9d0 msgr2=0x7f11d815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f11d802d9d0 0x7f11d815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 1 -- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 msgr2=0x7f11d80cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11d8038da0 0x7f11d80cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.102+0000 7f11de7fc640 1 -- 172.21.15.67:0/3657935820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d0009400 con 0x7f11d80c5790 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.103+0000 7fea377fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806f800 0x7fea38149b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57888/0 (socket says 172.21.15.67:57888) 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.103+0000 7fea377fe640 1 -- 172.21.15.67:0/3519114349 learned_addr learned my addr 172.21.15.67:0/3519114349 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.103+0000 7fea377fe640 10 monclient(hunting): get_auth_request con 0x7fea3806f800 auth_method 0 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.103+0000 7fea377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.103+0000 7fea377fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.103+0000 7fea377fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7f11c77fe640 10 client.?.objecter ms_handle_connect 0x7f11d80c5790 2023-12-01T17:04:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7f11c77fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7f11de7fc640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11d80c5790 0x7f11d815d240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f11d400c500 tx=0x7f11d400c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7fea36ffd640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea38061190 0x7fea38147860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7fea36ffd640 10 monclient(hunting): get_auth_request con 0x7fea38061190 auth_method 0 2023-12-01T17:04:04.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7fea36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7fea36ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.106+0000 7fea36ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.107+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.107+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.107+0000 7fea377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.107+0000 7fea36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.107+0000 7fea36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.107+0000 7fea36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 1 -- 172.21.15.67:0/3519114349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea3806efa0 msgr2=0x7fea38149650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fea3806efa0 0x7fea38149650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 1 -- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806f800 msgr2=0x7fea38149b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806f800 0x7fea38149b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea36ffd640 1 -- 172.21.15.67:0/3519114349 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea20009400 con 0x7fea38061190 2023-12-01T17:04:04.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.111+0000 7fea377fe640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fea3806f800 0x7fea38149b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-01T17:04:04.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.112+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f845401e7a0 con 0x7f845c13ba20 2023-12-01T17:04:04.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.113+0000 7f844affd640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f843c06c820 0x7f843c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.113+0000 7f845a59c640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f843c06c820 0x7f843c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.113+0000 7f845a59c640 10 monclient: get_auth_request con 0x7f843c06c820 auth_method 0 2023-12-01T17:04:04.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.113+0000 7f845a59c640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f843c06c820 0x7f843c06ece0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8450001560 tx=0x7f8450007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.113+0000 7f844affd640 10 client.4318.objecter ms_handle_connect 0x7f843c06c820 2023-12-01T17:04:04.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.114+0000 7fea36ffd640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea38061190 0x7fea38147860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c430 tx=0x7fea2800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.114+0000 7fea2ffff640 10 client.?.objecter ms_handle_connect 0x7fea38061190 2023-12-01T17:04:04.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.114+0000 7fea2ffff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.116+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8454059070 con 0x7f845c13ba20 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.116+0000 7f844affd640 10 client.4318.objecter ms_dispatch 0x7f845c000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.116+0000 7f844affd640 3 client.4318.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.116+0000 7f844affd640 3 client.4318.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 10 client.4318.objecter checking command 1 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f843c072a90 0x7f843c074eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 20 client.4318.objecter _get_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 20 client.4318.objecter put_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 15 client.4318.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 20 client.4318.objecter _get_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 20 client.4318.objecter get_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 15 client.4318.objecter _session_command_op_assign 6 1 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 20 client.4318.objecter put_session s=0x7f845400ce90 osd=6 4 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 10 client.4318.objecter _send_command 1 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 --> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f843c075560 con 0x7f843c072a90 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 20 client.4318.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8454058e40 con 0x7f845c13ba20 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f844affd640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.117+0000 7f845ad9d640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f843c072a90 0x7f843c074eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.118+0000 7f845ad9d640 10 monclient: get_auth_request con 0x7f843c072a90 auth_method 0 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.123+0000 7f845ad9d640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f843c072a90 0x7f843c074eb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.123+0000 7f844affd640 10 client.4318.objecter ms_handle_connect 0x7f843c072a90 2023-12-01T17:04:04.134 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.133+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== osd.5 v2:172.21.15.92:6800/1636405563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fbbdc075720 con 0x7fbbdc072c50 2023-12-01T17:04:04.134 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.133+0000 7fbbd77fe640 10 client.4309.objecter ms_dispatch 0x7fbbf0000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.135 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.133+0000 7fbbd77fe640 10 client.4309.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.135 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.133+0000 7fbbd77fe640 20 client.4309.objecter put_session s=0x7fbbdc072580 osd=5 4 2023-12-01T17:04:04.135 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.133+0000 7fbbd77fe640 15 client.4309.objecter _session_command_op_remove 5 1 2023-12-01T17:04:04.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 10 client.4309.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 15 client.4309.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 20 client.4309.objecter _get_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 20 client.4309.objecter put_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 20 client.4309.objecter _get_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 15 client.4309.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 20 client.4309.objecter get_session s=0x7fbbdc072580 osd=5 3 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 15 client.4309.objecter _session_command_op_assign 5 2 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 20 client.4309.objecter put_session s=0x7fbbdc072580 osd=5 4 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 10 client.4309.objecter _send_command 2 2023-12-01T17:04:04.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.148+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 --> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbbf00b4070 con 0x7fbbdc072c50 2023-12-01T17:04:04.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbd77fe640 1 -- 172.21.15.67:0/4145107819 <== osd.5 v2:172.21.15.92:6800/1636405563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbbe802c860 con 0x7fbbdc072c50 2023-12-01T17:04:04.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbd77fe640 10 client.4309.objecter ms_dispatch 0x7fbbf0000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbd77fe640 10 client.4309.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbd77fe640 20 client.4309.objecter put_session s=0x7fbbdc072580 osd=5 4 2023-12-01T17:04:04.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbd77fe640 15 client.4309.objecter _session_command_op_remove 5 2 2023-12-01T17:04:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 10 librados: watch_flush enter 2023-12-01T17:04:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 10 librados: watch_flush exit 2023-12-01T17:04:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 10 client.4309.objecter close_session for osd.5 2023-12-01T17:04:04.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fbbdc072c50 msgr2=0x7fbbdc075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fbbdc072c50 0x7fbbdc075070 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 20 client.4309.objecter put_session s=0x7fbbdc072580 osd=5 1 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 20 client.4309.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.155+0000 7fbbf5589640 10 client.4309.objecter successfully canceled tick 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbbdc06c870 msgr2=0x7fbbdc06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbbdc06c870 0x7fbbdc06ed30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00098f0 tx=0x7fbbe0008040 comp rx=0 tx=0).stop 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 10 monclient: shutdown 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 msgr2=0x7fbbf00abf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 --2- 172.21.15.67:0/4145107819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbbf00b2920 0x7fbbf00abf70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400b3e0 tx=0x7fbbe400b8a0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbef7fe640 1 -- 172.21.15.67:0/4145107819 reap_dead start 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 shutdown_connections 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 >> 172.21.15.67:0/4145107819 conn(0x7fbbf00b7890 msgr2=0x7fbbf0145a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 shutdown_connections 2023-12-01T17:04:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 -- 172.21.15.67:0/4145107819 wait complete. 2023-12-01T17:04:04.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.156+0000 7fbbf5589640 1 librados: shutdown 2023-12-01T17:04:04.169 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2023-12-01T17:04:04.170 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-01T17:04:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.176+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== osd.6 v2:172.21.15.92:6824/162801920 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f843c075560 con 0x7f843c072a90 2023-12-01T17:04:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.177+0000 7f844affd640 10 client.4318.objecter ms_dispatch 0x7f845c000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.177+0000 7f844affd640 10 client.4318.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.177+0000 7f844affd640 20 client.4318.objecter put_session s=0x7f845400ce90 osd=6 4 2023-12-01T17:04:04.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.177+0000 7f844affd640 15 client.4318.objecter _session_command_op_remove 6 1 2023-12-01T17:04:04.194 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.194 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 15 client.4318.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.194 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 20 client.4318.objecter _get_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.194 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 20 client.4318.objecter put_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.194 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 20 client.4318.objecter _get_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 15 client.4318.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 20 client.4318.objecter get_session s=0x7f845400ce90 osd=6 3 2023-12-01T17:04:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 15 client.4318.objecter _session_command_op_assign 6 2 2023-12-01T17:04:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 20 client.4318.objecter put_session s=0x7f845400ce90 osd=6 4 2023-12-01T17:04:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 10 client.4318.objecter _send_command 2 2023-12-01T17:04:04.195 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.193+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 --> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f845c13fb00 con 0x7f843c072a90 2023-12-01T17:04:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f844affd640 1 -- 172.21.15.67:0/2603768082 <== osd.6 v2:172.21.15.92:6824/162801920 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f844c031c40 con 0x7f843c072a90 2023-12-01T17:04:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f844affd640 10 client.4318.objecter ms_dispatch 0x7f845c000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f844affd640 10 client.4318.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f844affd640 20 client.4318.objecter put_session s=0x7f845400ce90 osd=6 4 2023-12-01T17:04:04.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f844affd640 15 client.4318.objecter _session_command_op_remove 6 2 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 10 librados: watch_flush enter 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 10 librados: watch_flush exit 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 10 client.4318.objecter close_session for osd.6 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f843c072a90 msgr2=0x7f843c074eb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f843c072a90 0x7f843c074eb0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 20 client.4318.objecter put_session s=0x7f845400ce90 osd=6 1 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 20 client.4318.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 10 client.4318.objecter successfully canceled tick 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.203+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f843c06c820 msgr2=0x7f843c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f843c06c820 0x7f843c06ece0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8450001560 tx=0x7f8450007990 comp rx=0 tx=0).stop 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 10 monclient: shutdown 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 msgr2=0x7f845c140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 --2- 172.21.15.67:0/2603768082 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f845c13ba20 0x7f845c140ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f845400bd70 tx=0x7f845400be60 comp rx=0 tx=0).stop 2023-12-01T17:04:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f845ad9d640 1 -- 172.21.15.67:0/2603768082 reap_dead start 2023-12-01T17:04:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 shutdown_connections 2023-12-01T17:04:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 >> 172.21.15.67:0/2603768082 conn(0x7f845c0b3b20 msgr2=0x7f845c0b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 shutdown_connections 2023-12-01T17:04:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 -- 172.21.15.67:0/2603768082 wait complete. 2023-12-01T17:04:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.204+0000 7f846094e640 1 librados: shutdown 2023-12-01T17:04:04.216 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2023-12-01T17:04:04.216 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-01T17:04:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.281+0000 7fe7fca46640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.281+0000 7fe7fca46640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.281+0000 7fe7fca46640 10 monclient: monmap: 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.283167+0000 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.283167+0000 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.283 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 1 Processor -- start 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 1 -- start start 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 10 monclient: init 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7f80a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f80b2280 0x7fe7f80a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f80b2280 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7f8145b00 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.282+0000 7fe7fca46640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f8145670 con 0x7fe7f8145b00 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.283+0000 7fe7fca46640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f80aa990 con 0x7fe7f80a8a20 2023-12-01T17:04:04.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.283+0000 7fe7fca46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.283+0000 7fe7fca46640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80aac70 con 0x7fe7f80b2280 2023-12-01T17:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.283+0000 7fe7fca46640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.283+0000 7fe7fca46640 10 monclient(hunting): authenticate will time out at 1375.521476s 2023-12-01T17:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.283+0000 7fe7f659c640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fe7f80a8a20 0x7fe7f813f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57786/0 (socket says 172.21.15.67:57786) 2023-12-01T17:04:04.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.283+0000 7fe7f659c640 1 -- 172.21.15.67:0/2863107555 learned_addr learned my addr 172.21.15.67:0/2863107555 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.286+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535709390 0 0) 0x7fe7f80aa990 con 0x7fe7f80a8a20 2023-12-01T17:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.286+0000 7fe7f559a640 10 monclient(hunting): _negotiate 2023-12-01T17:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.286+0000 7fe7f559a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.286+0000 7fe7f559a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.286+0000 7fe7f559a640 10 monclient(hunting): my global_id is 4327 2023-12-01T17:04:04.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.286+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7e0003520 con 0x7fe7f80a8a20 2023-12-01T17:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.287+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948212154 0 0) 0x7fe7f8145670 con 0x7fe7f8145b00 2023-12-01T17:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.287+0000 7fe7f559a640 10 monclient(hunting): _negotiate 2023-12-01T17:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.287+0000 7fe7f559a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.287+0000 7fe7f559a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.287+0000 7fe7f559a640 10 monclient(hunting): my global_id is 4278 2023-12-01T17:04:04.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.287+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7f80aa990 con 0x7fe7f8145b00 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 <== mon.1 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1608541516 0 0) 0x7fe7e0003520 con 0x7fe7f80a8a20 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f80b2280 msgr2=0x7fe7f80a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f80b2280 0x7fe7f80a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 >> v1:172.21.15.67:6789/0 conn(0x7fe7f8145b00 legacy=0x7fe7f814c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 monclient: found mon.noname-c-legacy 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.92:6789/0 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80aaf50 con 0x7fe7f80a8a20 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 <== mon.1 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3910818764 0 0) 0x7fe7e8001670 con 0x7fe7f80a8a20 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: dump: 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] but i'm connected to v1:172.21.15.92:6789/0, reconnecting 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 >> v1:172.21.15.92:6789/0 conn(0x7fe7f80a8a20 legacy=0x7fe7f813f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f80b2280 0x7fe7e0003b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): picked mon.c con 0x7fe7f80b2280 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e00040b0 0x7fe7e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): picked mon.a con 0x7fe7e00040b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0004e20 0x7fe7e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): picked mon.b con 0x7fe7e0004e20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f80aac70 con 0x7fe7e00040b0 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e0005640 con 0x7fe7e0004e20 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e000bef0 con 0x7fe7f80b2280 2023-12-01T17:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.289+0000 7fe7f559a640 10 monclient(hunting): _finish_auth 0 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f5d9b640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e00040b0 0x7fe7e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f6d9d640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0004e20 0x7fe7e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f5d9b640 10 monclient(hunting): get_auth_request con 0x7fe7e00040b0 auth_method 0 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f5d9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f5d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f6d9d640 10 monclient(hunting): get_auth_request con 0x7fe7e0004e20 auth_method 0 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f6d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.292+0000 7fe7f6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.293+0000 7fe7f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.293+0000 7fe7f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.293+0000 7fe7f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-01T17:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.293+0000 7fe7f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.293+0000 7fe7f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.293+0000 7fe7f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-01T17:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 10 monclient(hunting): handle_auth_done global_id 4327 payload 293 2023-12-01T17:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 1 -- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f80b2280 msgr2=0x7fe7e0003b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f80b2280 0x7fe7e0003b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 1 -- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e00040b0 msgr2=0x7fe7e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7e00040b0 0x7fe7e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 1 monclient: found mon.b 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 1 -- 172.21.15.67:0/2863107555 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e0003520 con 0x7fe7e0004e20 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7f6d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7fca46640 5 monclient: authenticate success, global_id 4327 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.295+0000 7fe7fca46640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.297+0000 7fe7f6d9d640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0004e20 0x7fe7e000b980 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001fd0 tx=0x7fe7e00057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1353e6a58547b1d1 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0003b70 con 0x7fe7e0004e20 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 10 monclient: dump: 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.302 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f0003cd0 con 0x7fe7e0004e20 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7fca46640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7fca46640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7fca46640 4 set_mon_vals no callback set 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 1 -- 172.21.15.67:0/2863107555 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00052f0 con 0x7fe7e0004e20 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.300+0000 7fe7f559a640 10 monclient: dump: 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.303 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.304 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.304 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 10 monclient: shutdown 2023-12-01T17:04:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 1 -- 172.21.15.67:0/2863107555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0004e20 msgr2=0x7fe7e000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 1 --2- 172.21.15.67:0/2863107555 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7e0004e20 0x7fe7e000b980 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0001fd0 tx=0x7fe7e00057f0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7f6d9d640 1 -- 172.21.15.67:0/2863107555 reap_dead start 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 1 -- 172.21.15.67:0/2863107555 shutdown_connections 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 1 -- 172.21.15.67:0/2863107555 >> 172.21.15.67:0/2863107555 conn(0x7fe7f8061a10 msgr2=0x7fe7f80b7880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 1 -- 172.21.15.67:0/2863107555 shutdown_connections 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.301+0000 7fe7fca46640 1 -- 172.21.15.67:0/2863107555 wait complete. 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 10 monclient: monmap: 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.307 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 1 librados: starting msgr at 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 1 librados: starting objecter 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 1 Processor -- start 2023-12-01T17:04:04.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 1 -- start start 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 1 librados: setting wanted keys 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 1 librados: calling monclient init 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.302+0000 7fe7fca46640 10 monclient: init 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7e0004e20 0x7fe7f813a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): picked mon.noname-c con 0x7fe7e0004e20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f81434a0 0x7fe7f8143920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f81434a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f8135bd0 0x7fe7f8136050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f8135bd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f813b820 con 0x7fe7f8135bd0 2023-12-01T17:04:04.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f813c3f0 con 0x7fe7f81434a0 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f813c6d0 con 0x7fe7e0004e20 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7fca46640 10 monclient(hunting): authenticate will time out at 1375.541768s 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7f5d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f81434a0 0x7fe7f8143920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.303+0000 7fe7f6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f8135bd0 0x7fe7f8136050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f81434a0 0x7fe7f8143920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36546/0 (socket says 172.21.15.67:36546) 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 1 -- 172.21.15.67:0/4065097718 learned_addr learned my addr 172.21.15.67:0/4065097718 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 10 monclient(hunting): get_auth_request con 0x7fe7f81434a0 auth_method 0 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f6d9d640 10 monclient(hunting): get_auth_request con 0x7fe7f8135bd0 auth_method 0 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f6d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.304+0000 7fe7f6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.305+0000 7fe7f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.305+0000 7fe7f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.305+0000 7fe7f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2023-12-01T17:04:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 1 -- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7e0004e20 msgr2=0x7fe7f813a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7e0004e20 0x7fe7f813a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 1 -- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f8135bd0 msgr2=0x7fe7f8136050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f8135bd0 0x7fe7f8136050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.307+0000 7fe7f5d9b640 1 -- 172.21.15.67:0/4065097718 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f813b300 con 0x7fe7f81434a0 2023-12-01T17:04:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.309+0000 7fe7f5d9b640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f81434a0 0x7fe7f8143920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00e840 tx=0x7fe7ec00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.309+0000 7fe7e6ffd640 10 client.?.objecter ms_handle_connect 0x7fe7f81434a0 2023-12-01T17:04:04.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.309+0000 7fe7e6ffd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 1 -- 172.21.15.67:0/4065097718 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec00c730 con 0x7fe7f81434a0 2023-12-01T17:04:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: dump: 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 1 -- 172.21.15.67:0/4065097718 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec00c890 con 0x7fe7f81434a0 2023-12-01T17:04:04.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 1 -- 172.21.15.67:0/4065097718 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec00cc40 con 0x7fe7f81434a0 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 5 monclient: authenticate success, global_id 4330 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 10 monclient: _renew_subs 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f8143e60 con 0x7fe7f81434a0 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7f4d99640 4 set_mon_vals no callback set 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7e6ffd640 10 monclient: dump: 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.316 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 10 monclient: _renew_subs 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f8136590 con 0x7fe7f81434a0 2023-12-01T17:04:04.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.313+0000 7fe7fca46640 1 librados: init done 2023-12-01T17:04:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.314+0000 7fe7fca46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.314+0000 7fe7fca46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.314+0000 7fe7fca46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.314+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f813b540 con 0x7fe7f81434a0 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 1 -- 172.21.15.67:0/466159458 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76002c980 con 0x7fe768071830 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: dump: 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.365 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 1 -- 172.21.15.67:0/466159458 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76002cae0 con 0x7fe768071830 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe76dcc2640 5 monclient: authenticate success, global_id 4259 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe76dcc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe76dcc2640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe76dcc2640 4 set_mon_vals no callback set 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 1 -- 172.21.15.67:0/466159458 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7600343f0 con 0x7fe768071830 2023-12-01T17:04:04.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.363+0000 7fe7667fc640 10 monclient: dump: 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.367 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 10 monclient: shutdown 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 -- 172.21.15.67:0/466159458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe768071830 msgr2=0x7fe768152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 --2- 172.21.15.67:0/466159458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe768071830 0x7fe768152440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe760009e50 tx=0x7fe76002bde0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 -- 172.21.15.67:0/466159458 shutdown_connections 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 --2- 172.21.15.67:0/466159458 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe768071830 0x7fe768152440 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 --2- 172.21.15.67:0/466159458 >> v2:172.21.15.67:3300/0 conn(0x7fe7680af630 0x7fe76806fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 -- 172.21.15.67:0/466159458 >> 172.21.15.67:0/466159458 conn(0x7fe7680b3b20 msgr2=0x7fe7680b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 -- 172.21.15.67:0/466159458 shutdown_connections 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.364+0000 7fe76dcc2640 1 -- 172.21.15.67:0/466159458 wait complete. 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.365+0000 7fe76dcc2640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.365+0000 7fe76dcc2640 10 monclient: monmap: 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.369 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fe76dcc2640 1 librados: starting msgr at 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fe76dcc2640 1 librados: starting objecter 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fe76dcc2640 1 Processor -- start 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fe76dcc2640 1 -- start start 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fe76dcc2640 1 librados: setting wanted keys 2023-12-01T17:04:04.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fe76dcc2640 1 librados: calling monclient init 2023-12-01T17:04:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fe76dcc2640 10 monclient: init 2023-12-01T17:04:04.371 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28005ce0 con 0x7fea38061190 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: dump: 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.373 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28005e40 con 0x7fea38061190 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 5 monclient: authenticate success, global_id 4274 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 10 monclient: _renew_subs 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 1 -- 172.21.15.67:0/3519114349 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea381470a0 con 0x7fea38061190 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28007310 con 0x7fea38061190 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea2ffff640 10 monclient: dump: 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 10 monclient: _renew_subs 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 1 -- 172.21.15.67:0/3519114349 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea38147da0 con 0x7fea38061190 2023-12-01T17:04:04.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea3d3ce640 1 librados: init done 2023-12-01T17:04:04.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7fea35ffb640 4 set_mon_vals no callback set 2023-12-01T17:04:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7f76727fc640 1 -- 172.21.15.67:0/991810357 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766802e8d0 con 0x7f7674072090 2023-12-01T17:04:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7f76727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7f76727fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.366+0000 7f76727fc640 10 monclient: dump: 2023-12-01T17:04:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.377 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 1 -- 172.21.15.67:0/991810357 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766802ea30 con 0x7f7674072090 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f767a02b640 5 monclient: authenticate success, global_id 4262 2023-12-01T17:04:04.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f767a02b640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 1 -- 172.21.15.67:0/991810357 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766802edc0 con 0x7f7674072090 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f767a02b640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f767a02b640 4 set_mon_vals no callback set 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe768071830 0x7fe76814e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): picked mon.noname-b con 0x7fe768071830 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7680af630 0x7fe7680abeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): picked mon.noname-c con 0x7fe7680af630 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe768152980 0x7fe768150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): picked mon.noname-a con 0x7fe768152980 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76814eab0 con 0x7fe768152980 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76814f800 con 0x7fe768071830 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe76814fa70 con 0x7fe7680af630 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fe76dcc2640 10 monclient(hunting): authenticate will time out at 1375.605781s 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fea3d3ce640 10 client.4274.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fea3d3ce640 15 client.4274.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.380 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fea3d3ce640 20 client.4274.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.381 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fea3d3ce640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.381 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fea3d3ce640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.381 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7fea3d3ce640 1 -- 172.21.15.67:0/3519114349 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fea38061570 con 0x7fea38061190 2023-12-01T17:04:04.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.382 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.367+0000 7f76727fc640 10 monclient: dump: 2023-12-01T17:04:04.382 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.382 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.382 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 10 monclient: shutdown 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 -- 172.21.15.67:0/991810357 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7674072090 msgr2=0x7f76740724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 --2- 172.21.15.67:0/991810357 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7674072090 0x7f76740724b0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f767406cde0 tx=0x7f766802cb10 comp rx=0 tx=0).stop 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 -- 172.21.15.67:0/991810357 shutdown_connections 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 --2- 172.21.15.67:0/991810357 >> v2:172.21.15.92:3300/0 conn(0x7f767413b4c0 0x7f767413b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 --2- 172.21.15.67:0/991810357 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7674072090 0x7f76740724b0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 --2- 172.21.15.67:0/991810357 >> v2:172.21.15.67:3300/0 conn(0x7f76740af380 0x7f767406fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.384 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 -- 172.21.15.67:0/991810357 >> 172.21.15.67:0/991810357 conn(0x7f76740b3ef0 msgr2=0x7f76740b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.384 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 -- 172.21.15.67:0/991810357 shutdown_connections 2023-12-01T17:04:04.384 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7fe767fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe768152980 0x7fe768150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.384 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7f767a02b640 1 -- 172.21.15.67:0/991810357 wait complete. 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7fe767fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe768152980 0x7fe768150170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57912/0 (socket says 172.21.15.67:57912) 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7fe767fff640 1 -- 172.21.15.67:0/1504842160 learned_addr learned my addr 172.21.15.67:0/1504842160 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7fe767fff640 10 monclient(hunting): get_auth_request con 0x7fe768152980 auth_method 0 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7fe767fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7fe767fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.368+0000 7fe767fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.369+0000 7fe767fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.369+0000 7fe767fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.369+0000 7fe767fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.369+0000 7f767a02b640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.369+0000 7f767a02b640 10 monclient: monmap: 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.370+0000 7f767a02b640 1 librados: starting msgr at 2023-12-01T17:04:04.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.370+0000 7f767a02b640 1 librados: starting objecter 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.370+0000 7f767a02b640 1 Processor -- start 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.370+0000 7f767a02b640 1 -- start start 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.370+0000 7f767a02b640 1 librados: setting wanted keys 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.370+0000 7f767a02b640 1 librados: calling monclient init 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.370+0000 7f767a02b640 10 monclient: init 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.371+0000 7f767a02b640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.371+0000 7f767a02b640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.371+0000 7f767a02b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7674072090 0x7f76741d63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.371+0000 7f767a02b640 10 monclient(hunting): picked mon.noname-a con 0x7f7674072090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.371+0000 7f767a02b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f76740af380 0x7f76741d6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 10 monclient(hunting): picked mon.noname-c con 0x7f76740af380 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f767413b4c0 0x7f76741c5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 10 monclient(hunting): picked mon.noname-b con 0x7f767413b4c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76741d6e60 con 0x7f7674072090 2023-12-01T17:04:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76741c5670 con 0x7f767413b4c0 2023-12-01T17:04:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76741c5830 con 0x7f76740af380 2023-12-01T17:04:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7f767a02b640 10 monclient(hunting): authenticate will time out at 1375.610671s 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 1 -- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7680af630 msgr2=0x7fe7680abeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7680af630 0x7fe7680abeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 1 -- 172.21.15.67:0/1504842160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe768071830 msgr2=0x7fe76814e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe768071830 0x7fe76814e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 1 monclient: found mon.noname-a 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.391 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.372+0000 7fe767fff640 1 -- 172.21.15.67:0/1504842160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe760009b30 con 0x7fe768152980 2023-12-01T17:04:04.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4005ce0 con 0x7f11d80c5790 2023-12-01T17:04:04.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: dump: 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.393 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d4005e40 con 0x7f11d80c5790 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 5 monclient: authenticate success, global_id 4271 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 10 monclient: _renew_subs 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d80cbd70 con 0x7f11d80c5790 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d4005ce0 con 0x7f11d80c5790 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11c77fe640 10 monclient: dump: 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.394 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 10 monclient: _renew_subs 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d802f450 con 0x7f11d80c5790 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dffff640 1 librados: init done 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.373+0000 7f11dd7fa640 4 set_mon_vals no callback set 2023-12-01T17:04:04.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7f11dffff640 10 client.4271.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7f11dffff640 15 client.4271.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7f11dffff640 20 client.4271.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7f11dffff640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7f11dffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f11d80c8ca0 con 0x7f11d80c5790 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7fd67affd640 1 -- 172.21.15.67:0/835960555 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c031890 con 0x7fd67c146d70 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7fd67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7fd67affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.374+0000 7fd67affd640 10 monclient: dump: 2023-12-01T17:04:04.396 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 5 monclient: authenticate success, global_id 4256 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 1 -- 172.21.15.67:0/835960555 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66c0319f0 con 0x7fd67c146d70 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 4 set_mon_vals no callback set 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 1 -- 172.21.15.67:0/835960555 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66c031d80 con 0x7fd67c146d70 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd67affd640 10 monclient: dump: 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.398 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.399 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.399 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fe767fff640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe768152980 0x7fe768150170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00b4c0 tx=0x7fe75c00b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fe74ffff640 10 client.?.objecter ms_handle_connect 0x7fe768152980 2023-12-01T17:04:04.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fe74ffff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 10 monclient: shutdown 2023-12-01T17:04:04.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 1 -- 172.21.15.67:0/835960555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c146d70 msgr2=0x7fd67c149ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 1 --2- 172.21.15.67:0/835960555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c146d70 0x7fd67c149ae0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0097d0 tx=0x7fd66c02b930 comp rx=0 tx=0).stop 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.375+0000 7fd6817b3640 1 -- 172.21.15.67:0/835960555 shutdown_connections 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.376+0000 7fd6817b3640 1 --2- 172.21.15.67:0/835960555 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c146d70 0x7fd67c149ae0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.376+0000 7fd6817b3640 1 -- 172.21.15.67:0/835960555 >> 172.21.15.67:0/835960555 conn(0x7fd67c06c0c0 msgr2=0x7fd67c0b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.376+0000 7fd6817b3640 1 -- 172.21.15.67:0/835960555 shutdown_connections 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.376+0000 7fd6817b3640 1 -- 172.21.15.67:0/835960555 wait complete. 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.376+0000 7fd6817b3640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.376+0000 7fd6817b3640 10 monclient: monmap: 2023-12-01T17:04:04.401 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.402 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.377+0000 7fd6817b3640 1 librados: starting msgr at 2023-12-01T17:04:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.377+0000 7fd6817b3640 1 librados: starting objecter 2023-12-01T17:04:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.377+0000 7fd6817b3640 1 Processor -- start 2023-12-01T17:04:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.377+0000 7fd6817b3640 1 -- start start 2023-12-01T17:04:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.377+0000 7fd6817b3640 1 librados: setting wanted keys 2023-12-01T17:04:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.377+0000 7fd6817b3640 1 librados: calling monclient init 2023-12-01T17:04:04.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.377+0000 7fd6817b3640 10 monclient: init 2023-12-01T17:04:04.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7f84d6df0640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7f84d6df0640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7f84d6df0640 10 monclient: monmap: 2023-12-01T17:04:04.404 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.404 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.404 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.379988+0000 2023-12-01T17:04:04.404 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.379988+0000 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.405 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c06a770 0x7fd67c140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): picked mon.noname-c con 0x7fd67c06a770 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd67c14a020 0x7fd67c143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): picked mon.noname-b con 0x7fd67c14a020 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c145d60 0x7fd67c144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): picked mon.noname-a con 0x7fd67c145d60 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c141a20 con 0x7fd67c145d60 2023-12-01T17:04:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c149100 con 0x7fd67c14a020 2023-12-01T17:04:04.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67c11c300 con 0x7fd67c06a770 2023-12-01T17:04:04.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.378+0000 7fd6817b3640 10 monclient(hunting): authenticate will time out at 1375.616798s 2023-12-01T17:04:04.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f84d6df0640 1 Processor -- start 2023-12-01T17:04:04.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 1 -- 172.21.15.67:0/3782567203 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e402e070 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: dump: 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.411 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04fadd2640 5 monclient: authenticate success, global_id 4268 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04fadd2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.412 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 1 -- 172.21.15.67:0/3782567203 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04e40315a0 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04fadd2640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04fadd2640 4 set_mon_vals no callback set 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 1 -- 172.21.15.67:0/3782567203 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04e402e070 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f04f8dce640 10 monclient: dump: 2023-12-01T17:04:04.413 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.414 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.415 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.415 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f84d6df0640 1 -- start start 2023-12-01T17:04:04.415 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.379+0000 7f84d6df0640 10 monclient: init 2023-12-01T17:04:04.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 10 monclient: shutdown 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 -- 172.21.15.67:0/3782567203 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec0c7fd0 msgr2=0x7f04ec0c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 --2- 172.21.15.67:0/3782567203 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec0c7fd0 0x7f04ec0c8410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f04e40097a0 tx=0x7f04e402b840 comp rx=0 tx=0).stop 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 -- 172.21.15.67:0/3782567203 shutdown_connections 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 --2- 172.21.15.67:0/3782567203 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec0c7fd0 0x7f04ec0c8410 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 --2- 172.21.15.67:0/3782567203 >> v2:172.21.15.67:3300/0 conn(0x7f04ec0c5560 0x7f04ec007210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 -- 172.21.15.67:0/3782567203 >> 172.21.15.67:0/3782567203 conn(0x7f04ec0062d0 msgr2=0x7f04ec0066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 -- 172.21.15.67:0/3782567203 shutdown_connections 2023-12-01T17:04:04.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f04fadd2640 1 -- 172.21.15.67:0/3782567203 wait complete. 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f84d0071ee0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f84d00722c0 0x7f84d0141820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 10 monclient(hunting): picked mon.noname-a con 0x7f84d00722c0 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d0141ea0 0x7f84d0142310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 10 monclient(hunting): picked mon.noname-b con 0x7f84d0141ea0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.380+0000 7f84d6df0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84d0071500 con 0x7f84d0071ee0 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f84d6df0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f84d6df0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84d0070410 con 0x7f84d00722c0 2023-12-01T17:04:04.420 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f84d6df0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.420 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f84d6df0640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d006f9a0 con 0x7f84d0141ea0 2023-12-01T17:04:04.420 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f84d6df0640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.420 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f84d6df0640 10 monclient(hunting): authenticate will time out at 1375.619513s 2023-12-01T17:04:04.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f04fadd2640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.381+0000 7f04fadd2640 10 monclient: monmap: 2023-12-01T17:04:04.422 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.422 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.422 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.382+0000 7f04fadd2640 1 librados: starting msgr at 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.382+0000 7f04fadd2640 1 librados: starting objecter 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.382+0000 7f04fadd2640 1 Processor -- start 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.382+0000 7f04fadd2640 1 -- start start 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.382+0000 7f04fadd2640 1 librados: setting wanted keys 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.382+0000 7f04fadd2640 1 librados: calling monclient init 2023-12-01T17:04:04.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.382+0000 7f04fadd2640 10 monclient: init 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.383+0000 7f04fadd2640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.383+0000 7f04fadd2640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.383+0000 7f04fadd2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec036bf0 0x7f04ec034910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.383+0000 7f04fadd2640 10 monclient(hunting): picked mon.noname-c con 0x7f04ec036bf0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04ec0c5560 0x7f04ec0a3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 10 monclient(hunting): picked mon.noname-b con 0x7f04ec0c5560 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0c7fd0 0x7f04ec160660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 10 monclient(hunting): picked mon.noname-a con 0x7f04ec0c7fd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec160180 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec0a4610 con 0x7f04ec0c5560 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec031c70 con 0x7f04ec036bf0 2023-12-01T17:04:04.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.384+0000 7f04fadd2640 10 monclient(hunting): authenticate will time out at 1375.622947s 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c011830 con 0x7fe768152980 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe74ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe74ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe74ffff640 10 monclient: dump: 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.425 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe74ffff640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe74ffff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe74ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 5 monclient: authenticate success, global_id 4284 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 10 monclient: _renew_subs 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7680ac3f0 con 0x7fe768152980 2023-12-01T17:04:04.427 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f11d40070f0 con 0x7f11d80c5790 2023-12-01T17:04:04.427 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 10 monclient: _renew_subs 2023-12-01T17:04:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76814d680 con 0x7fe768152980 2023-12-01T17:04:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7fe76dcc2640 1 librados: init done 2023-12-01T17:04:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7f11c77fe640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f11cc06c8e0 0x7f11cc06eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.428 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7f11deffd640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f11cc06c8e0 0x7f11cc06eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.404+0000 7f11deffd640 10 monclient: get_auth_request con 0x7f11cc06c8e0 auth_method 0 2023-12-01T17:04:04.429 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7fe76dcc2640 10 client.4284.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7fe76dcc2640 15 client.4284.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7fe76dcc2640 20 client.4284.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7fe76dcc2640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7fe76dcc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe76814fec0 con 0x7fe768152980 2023-12-01T17:04:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7f11deffd640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f11cc06c8e0 0x7f11cc06eda0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11d0036000 tx=0x7f11d0008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.405+0000 7f11c77fe640 10 client.4271.objecter ms_handle_connect 0x7f11cc06c8e0 2023-12-01T17:04:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f11d4057dc0 con 0x7f11d80c5790 2023-12-01T17:04:04.431 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 10 client.4271.objecter ms_dispatch 0x7f11e0000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 3 client.4271.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 3 client.4271.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 10 client.4271.objecter checking command 1 2023-12-01T17:04:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f11cc072cc0 0x7f11cc0750e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 20 client.4271.objecter _get_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 20 client.4271.objecter put_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe75c009170 con 0x7fe768152980 2023-12-01T17:04:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7fe74ffff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 15 client.4271.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 20 client.4271.objecter _get_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 20 client.4271.objecter get_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 15 client.4271.objecter _session_command_op_assign 3 1 2023-12-01T17:04:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 20 client.4271.objecter put_session s=0x7f11cc0725f0 osd=3 4 2023-12-01T17:04:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 10 client.4271.objecter _send_command 1 2023-12-01T17:04:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 --> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f11cc075790 con 0x7f11cc072cc0 2023-12-01T17:04:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 20 client.4271.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c011b90 con 0x7fe768152980 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7fe74ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7fe74ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7fe74ffff640 10 monclient: dump: 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.435 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.436 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.436 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11d4005ce0 con 0x7f11d80c5790 2023-12-01T17:04:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7f11c77fe640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.408+0000 7fe765ffb640 4 set_mon_vals no callback set 2023-12-01T17:04:04.437 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.409+0000 7f11df7fe640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f11cc072cc0 0x7f11cc0750e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.438 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.409+0000 7f11df7fe640 10 monclient: get_auth_request con 0x7f11cc072cc0 auth_method 0 2023-12-01T17:04:04.438 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.413+0000 7f11c77fe640 10 client.4271.objecter ms_handle_connect 0x7f11cc072cc0 2023-12-01T17:04:04.438 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.413+0000 7f11df7fe640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f11cc072cc0 0x7f11cc0750e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.438 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.414+0000 7f7672ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f76740af380 0x7f76741d6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.415+0000 7f84d538d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d0141ea0 0x7f84d0142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.416+0000 7f84d538d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d0141ea0 0x7f84d0142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41890/0 (socket says 172.21.15.67:41890) 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.416+0000 7f84d538d640 1 -- 172.21.15.67:0/3914509024 learned_addr learned my addr 172.21.15.67:0/3914509024 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.416+0000 7f84d538d640 10 monclient(hunting): get_auth_request con 0x7f84d0141ea0 auth_method 0 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.416+0000 7f84d538d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.416+0000 7f84d538d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.416+0000 7f84d538d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.417+0000 7f84d538d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.417+0000 7f84d538d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.417+0000 7f84d538d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.418+0000 7f7672ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f76740af380 0x7f76741d6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41870/0 (socket says 172.21.15.67:41870) 2023-12-01T17:04:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.418+0000 7f7672ffd640 1 -- 172.21.15.67:0/3391583898 learned_addr learned my addr 172.21.15.67:0/3391583898 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.418+0000 7f7672ffd640 10 monclient(hunting): get_auth_request con 0x7f76740af380 auth_method 0 2023-12-01T17:04:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.418+0000 7f7672ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.418+0000 7f7672ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.418+0000 7f7672ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.418+0000 7fd680fb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c145d60 0x7fd67c144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 10 monclient(hunting): handle_auth_done global_id 4277 payload 274 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 1 -- 172.21.15.67:0/3914509024 >> v2:172.21.15.67:3300/0 conn(0x7f84d00722c0 msgr2=0x7f84d0141820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 1 --2- 172.21.15.67:0/3914509024 >> v2:172.21.15.67:3300/0 conn(0x7f84d00722c0 0x7f84d0141820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 1 -- 172.21.15.67:0/3914509024 >> v1:172.21.15.67:6789/0 conn(0x7f84d0071ee0 legacy=0x7f84d006f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.420+0000 7f84d538d640 1 -- 172.21.15.67:0/3914509024 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d0071750 con 0x7f84d0141ea0 2023-12-01T17:04:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.427+0000 7f7672ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.427+0000 7f7672ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.427+0000 7f7672ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.429+0000 7fd680fb2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c145d60 0x7fd67c144490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57930/0 (socket says 172.21.15.67:57930) 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.429+0000 7fd680fb2640 1 -- 172.21.15.67:0/874797375 learned_addr learned my addr 172.21.15.67:0/874797375 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.429+0000 7fd680fb2640 10 monclient(hunting): get_auth_request con 0x7fd67c145d60 auth_method 0 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.429+0000 7fd680fb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.429+0000 7fd680fb2640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.429+0000 7fd680fb2640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.430+0000 7fd680fb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.430+0000 7fd680fb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.430+0000 7fd680fb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2023-12-01T17:04:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 1 -- 172.21.15.67:0/3391583898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f767413b4c0 msgr2=0x7f76741c5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f767413b4c0 0x7f76741c5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 1 -- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7674072090 msgr2=0x7f76741d63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7674072090 0x7f76741d63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.435+0000 7f7672ffd640 1 -- 172.21.15.67:0/3391583898 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76680094d0 con 0x7f76740af380 2023-12-01T17:04:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.444+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe75c011cf0 con 0x7fe768152980 2023-12-01T17:04:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.444+0000 7fe74ffff640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe75406c820 0x7fe75406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.445+0000 7fe7677fe640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe75406c820 0x7fe75406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.445+0000 7fe7677fe640 10 monclient: get_auth_request con 0x7fe75406c820 auth_method 0 2023-12-01T17:04:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.445+0000 7fe74ffff640 10 client.4284.objecter ms_handle_connect 0x7fe75406c820 2023-12-01T17:04:04.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.445+0000 7fe7677fe640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe75406c820 0x7fe75406ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe758009710 tx=0x7fe758006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe75c00f2e0 con 0x7fe768152980 2023-12-01T17:04:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 10 client.4284.objecter ms_dispatch 0x7fe768000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 3 client.4284.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 3 client.4284.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 10 client.4284.objecter checking command 1 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7fe754072c00 0x7fe754075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 20 client.4284.objecter _get_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 20 client.4284.objecter put_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 15 client.4284.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 20 client.4284.objecter _get_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 20 client.4284.objecter get_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 15 client.4284.objecter _session_command_op_assign 2 1 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 20 client.4284.objecter put_session s=0x7fe754072530 osd=2 4 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 10 client.4284.objecter _send_command 1 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe7540756d0 con 0x7fe754072c00 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 20 client.4284.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe75c054d90 con 0x7fe768152980 2023-12-01T17:04:04.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.449+0000 7fe74ffff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.450+0000 7fe766ffd640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7fe754072c00 0x7fe754075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.450+0000 7fe766ffd640 10 monclient: get_auth_request con 0x7fe754072c00 auth_method 0 2023-12-01T17:04:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.455+0000 7fe766ffd640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7fe754072c00 0x7fe754075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.455+0000 7fe74ffff640 10 client.4284.objecter ms_handle_connect 0x7fe754072c00 2023-12-01T17:04:04.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2023-12-01T17:04:04.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 1 -- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c06a770 msgr2=0x7fd67c140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd67c06a770 0x7fd67c140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 1 -- 172.21.15.67:0/874797375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd67c14a020 msgr2=0x7fd67c143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd67c14a020 0x7fd67c143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7f84d538d640 1 --2- 172.21.15.67:0/3914509024 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d0141ea0 0x7f84d0142310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009e50 tx=0x7f84c802be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31e84912ca8a8efc server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 1 monclient: found mon.noname-a 2023-12-01T17:04:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fd680fb2640 1 -- 172.21.15.67:0/874797375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c009400 con 0x7fd67c145d60 2023-12-01T17:04:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fea28007470 con 0x7fea38061190 2023-12-01T17:04:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.457+0000 7fea2ffff640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fea1806c870 0x7fea1806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.458+0000 7fea2e7fc640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fea1806c870 0x7fea1806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.458+0000 7fea2e7fc640 10 monclient: get_auth_request con 0x7fea1806c870 auth_method 0 2023-12-01T17:04:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.458+0000 7fea2e7fc640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fea1806c870 0x7fea1806ed30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea20035000 tx=0x7fea20030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.458+0000 7fea2ffff640 10 client.4274.objecter ms_handle_connect 0x7fea1806c870 2023-12-01T17:04:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fd680fb2640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c145d60 0x7fd67c144490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd67000c3c0 tx=0x7fd670001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fd678ff9640 10 client.?.objecter ms_handle_connect 0x7fd67c145d60 2023-12-01T17:04:04.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fd678ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fea28007720 con 0x7fea38061190 2023-12-01T17:04:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 10 client.4274.objecter ms_dispatch 0x7fea38000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 3 client.4274.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 3 client.4274.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 10 client.4274.objecter checking command 1 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fea180735f0 0x7fea18075a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 20 client.4274.objecter _get_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 20 client.4274.objecter put_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 15 client.4274.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 20 client.4274.objecter _get_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 20 client.4274.objecter get_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 15 client.4274.objecter _session_command_op_assign 7 1 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 20 client.4274.objecter put_session s=0x7fea18072f20 osd=7 4 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 10 client.4274.objecter _send_command 1 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 --> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fea18076160 con 0x7fea180735f0 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 20 client.4274.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea28057ec0 con 0x7fea38061190 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.461+0000 7fea2ffff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.462+0000 7fea377fe640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fea180735f0 0x7fea18075a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.462+0000 7fea377fe640 10 monclient: get_auth_request con 0x7fea180735f0 auth_method 0 2023-12-01T17:04:04.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.464+0000 7f04fa5d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0c7fd0 0x7f04ec160660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.464+0000 7f7672ffd640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f76740af380 0x7f76741d6920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f76680090b0 tx=0x7f76680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.464+0000 7f765bfff640 10 client.?.objecter ms_handle_connect 0x7f76740af380 2023-12-01T17:04:04.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.464+0000 7f765bfff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.466+0000 7fea2ffff640 10 client.4274.objecter ms_handle_connect 0x7fea180735f0 2023-12-01T17:04:04.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.466+0000 7fea377fe640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fea180735f0 0x7fea18075a90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84cf7fe640 1 -- 172.21.15.67:0/3914509024 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84c802c9e0 con 0x7f84d0141ea0 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84cf7fe640 10 monclient: dump: 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.469 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.470 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84cf7fe640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84cf7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84d6df0640 5 monclient: authenticate success, global_id 4277 2023-12-01T17:04:04.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.467+0000 7f84d6df0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.469+0000 7f04fa5d1640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0c7fd0 0x7f04ec160660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57944/0 (socket says 172.21.15.67:57944) 2023-12-01T17:04:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.469+0000 7f04fa5d1640 1 -- 172.21.15.67:0/1221287159 learned_addr learned my addr 172.21.15.67:0/1221287159 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.469+0000 7f04fa5d1640 10 monclient(hunting): get_auth_request con 0x7f04ec0c7fd0 auth_method 0 2023-12-01T17:04:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.469+0000 7f04fa5d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.469+0000 7f04fa5d1640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.469+0000 7f04fa5d1640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.469+0000 7f04f9dd0640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec036bf0 0x7f04ec034910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670005ce0 con 0x7fd67c145d60 2023-12-01T17:04:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: dump: 2023-12-01T17:04:04.472 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 5 monclient: authenticate success, global_id 4287 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 10 monclient: _renew_subs 2023-12-01T17:04:04.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 1 -- 172.21.15.67:0/874797375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd67c141500 con 0x7fd67c145d60 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd670005e40 con 0x7fd67c145d60 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670007310 con 0x7fd67c145d60 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd678ff9640 10 monclient: dump: 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.474 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 10 monclient: _renew_subs 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 1 -- 172.21.15.67:0/874797375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd67c1493e0 con 0x7fd67c145d60 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd6817b3640 1 librados: init done 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.470+0000 7fd679ffb640 4 set_mon_vals no callback set 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7f04fa5d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7f04fa5d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7f04fa5d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7fd6817b3640 10 client.4287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7fd6817b3640 15 client.4287.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7fd6817b3640 20 client.4287.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7fd6817b3640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7fd6817b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.471+0000 7fd6817b3640 1 -- 172.21.15.67:0/874797375 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd67c0b0220 con 0x7fd67c145d60 2023-12-01T17:04:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f04f9dd0640 10 monclient(hunting): get_auth_request con 0x7f04ec036bf0 auth_method 0 2023-12-01T17:04:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f04f9dd0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f04f9dd0640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f04f9dd0640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f84cf7fe640 1 -- 172.21.15.67:0/3914509024 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84c802cb40 con 0x7f84d0141ea0 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f84cf7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f84d6df0640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.473+0000 7f84d6df0640 4 set_mon_vals no callback set 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 10 monclient: shutdown 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 1 -- 172.21.15.67:0/3914509024 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d0141ea0 msgr2=0x7f84d0142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 1 --2- 172.21.15.67:0/3914509024 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d0141ea0 0x7f84d0142310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f84c8009e50 tx=0x7f84c802be40 comp rx=0 tx=0).stop 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 1 -- 172.21.15.67:0/3914509024 shutdown_connections 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 1 --2- 172.21.15.67:0/3914509024 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d0141ea0 0x7f84d0142310 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.478 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 1 --2- 172.21.15.67:0/3914509024 >> v2:172.21.15.67:3300/0 conn(0x7f84d00722c0 0x7f84d0141820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.474+0000 7f84d6df0640 1 -- 172.21.15.67:0/3914509024 >> 172.21.15.67:0/3914509024 conn(0x7f84d00b3e50 msgr2=0x7f84d00b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f84d6df0640 1 -- 172.21.15.67:0/3914509024 shutdown_connections 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f84d6df0640 1 -- 172.21.15.67:0/3914509024 wait complete. 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec036bf0 msgr2=0x7f04ec034910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f04ec036bf0 0x7f04ec034910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04ec0c5560 msgr2=0x7f04ec0a3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f04ec0c5560 0x7f04ec0a3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 1 monclient: found mon.noname-a 2023-12-01T17:04:04.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f04fa5d1640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e4009400 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== osd.3 v2:172.21.15.67:6802/1625009733 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f11cc075790 con 0x7f11cc072cc0 2023-12-01T17:04:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f11c77fe640 10 client.4271.objecter ms_dispatch 0x7f11e0000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f11c77fe640 10 client.4271.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f11c77fe640 20 client.4271.objecter put_session s=0x7f11cc0725f0 osd=3 4 2023-12-01T17:04:04.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.475+0000 7f11c77fe640 15 client.4271.objecter _session_command_op_remove 3 1 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.476+0000 7f84d6df0640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.476+0000 7f84d6df0640 10 monclient: monmap: 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.476+0000 7f84d6df0640 1 librados: starting msgr at 2023-12-01T17:04:04.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.476+0000 7f84d6df0640 1 librados: starting objecter 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.476+0000 7f84d6df0640 1 Processor -- start 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.477+0000 7f84d6df0640 1 -- start start 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.477+0000 7f84d6df0640 1 librados: setting wanted keys 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.477+0000 7f84d6df0640 1 librados: calling monclient init 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.477+0000 7f84d6df0640 10 monclient: init 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.477+0000 7f84d6df0640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.477+0000 7f84d6df0640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.477+0000 7f84d6df0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d0071ee0 0x7f84d01391a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): picked mon.noname-a con 0x7f84d0071ee0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d00722c0 0x7f84d00acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): picked mon.noname-c con 0x7f84d00722c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d0141ea0 0x7f84d0137d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): picked mon.noname-b con 0x7f84d0141ea0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0138d40 con 0x7f84d0071ee0 2023-12-01T17:04:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0138930 con 0x7f84d0141ea0 2023-12-01T17:04:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84d0138270 con 0x7f84d00722c0 2023-12-01T17:04:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.478+0000 7f84d6df0640 10 monclient(hunting): authenticate will time out at 1375.717082s 2023-12-01T17:04:04.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.481+0000 7f84cffff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d00722c0 0x7f84d00acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766803f420 con 0x7f76740af380 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: dump: 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.486 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f766803f5c0 con 0x7f76740af380 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 5 monclient: authenticate success, global_id 4280 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f766803f970 con 0x7f76740af380 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f765bfff640 10 monclient: dump: 2023-12-01T17:04:04.487 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 10 monclient: _renew_subs 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76741c5ab0 con 0x7f76740af380 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f7671ffb640 4 set_mon_vals no callback set 2023-12-01T17:04:04.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 10 monclient: _renew_subs 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7674134470 con 0x7f76740af380 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f767a02b640 1 librados: init done 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f84cffff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d00722c0 0x7f84d00acb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41912/0 (socket says 172.21.15.67:41912) 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f84cffff640 1 -- 172.21.15.67:0/531762456 learned_addr learned my addr 172.21.15.67:0/531762456 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f84cffff640 10 monclient(hunting): get_auth_request con 0x7f84d00722c0 auth_method 0 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f84cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f84cffff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.484+0000 7f84cffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f84cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f767a02b640 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f767a02b640 15 client.4280.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f767a02b640 20 client.4280.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f767a02b640 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f767a02b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.485+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7674070ab0 con 0x7f76740af380 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 1 -- 172.21.15.67:0/531762456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d0141ea0 msgr2=0x7f84d0137d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d0141ea0 0x7f84d0137d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 1 -- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d0071ee0 msgr2=0x7f84d01391a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d0071ee0 0x7f84d01391a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.488+0000 7f84cffff640 1 -- 172.21.15.67:0/531762456 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c8009b30 con 0x7f84d00722c0 2023-12-01T17:04:04.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 10 client.4271.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 15 client.4271.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 20 client.4271.objecter _get_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 20 client.4271.objecter put_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 20 client.4271.objecter _get_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 15 client.4271.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 20 client.4271.objecter get_session s=0x7f11cc0725f0 osd=3 3 2023-12-01T17:04:04.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 15 client.4271.objecter _session_command_op_assign 3 2 2023-12-01T17:04:04.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 20 client.4271.objecter put_session s=0x7f11cc0725f0 osd=3 4 2023-12-01T17:04:04.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 10 client.4271.objecter _send_command 2 2023-12-01T17:04:04.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.495+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 --> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f11d815cf40 con 0x7f11cc072cc0 2023-12-01T17:04:04.501 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.500+0000 7f84cffff640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d00722c0 0x7f84d00acb40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00b4d0 tx=0x7f84bc00b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.500+0000 7f84cd7fa640 10 client.?.objecter ms_handle_connect 0x7f84d00722c0 2023-12-01T17:04:04.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.500+0000 7f84cd7fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11c77fe640 1 -- 172.21.15.67:0/3657935820 <== osd.3 v2:172.21.15.67:6802/1625009733 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f11d815cf40 con 0x7f11cc072cc0 2023-12-01T17:04:04.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11c77fe640 10 client.4271.objecter ms_dispatch 0x7f11e0000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11c77fe640 10 client.4271.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11c77fe640 20 client.4271.objecter put_session s=0x7f11cc0725f0 osd=3 4 2023-12-01T17:04:04.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11c77fe640 15 client.4271.objecter _session_command_op_remove 3 2 2023-12-01T17:04:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11dffff640 10 librados: watch_flush enter 2023-12-01T17:04:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11dffff640 10 librados: watch_flush exit 2023-12-01T17:04:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11dffff640 10 client.4271.objecter close_session for osd.3 2023-12-01T17:04:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f11cc072cc0 msgr2=0x7f11cc0750e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11dffff640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f11cc072cc0 0x7f11cc0750e0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11dffff640 20 client.4271.objecter put_session s=0x7f11cc0725f0 osd=3 1 2023-12-01T17:04:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.507+0000 7f11dffff640 20 client.4271.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 10 client.4271.objecter successfully canceled tick 2023-12-01T17:04:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f11cc06c8e0 msgr2=0x7f11cc06eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f11cc06c8e0 0x7f11cc06eda0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f11d0036000 tx=0x7f11d0008030 comp rx=0 tx=0).stop 2023-12-01T17:04:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 10 monclient: shutdown 2023-12-01T17:04:04.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd670007470 con 0x7fd67c145d60 2023-12-01T17:04:04.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11d80c5790 msgr2=0x7f11d815d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 1 --2- 172.21.15.67:0/3657935820 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f11d80c5790 0x7f11d815d240 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f11d400c500 tx=0x7f11d400c9c0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11df7fe640 1 -- 172.21.15.67:0/3657935820 reap_dead start 2023-12-01T17:04:04.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 shutdown_connections 2023-12-01T17:04:04.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 >> 172.21.15.67:0/3657935820 conn(0x7f11d8006600 msgr2=0x7f11d80384e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7fd678ff9640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd66806c820 0x7fd66806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.508+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 shutdown_connections 2023-12-01T17:04:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.509+0000 7f11dffff640 1 -- 172.21.15.67:0/3657935820 wait complete. 2023-12-01T17:04:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.509+0000 7fd67bfff640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd66806c820 0x7fd66806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.509+0000 7f11dffff640 1 librados: shutdown 2023-12-01T17:04:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.509+0000 7fd67bfff640 10 monclient: get_auth_request con 0x7fd66806c820 auth_method 0 2023-12-01T17:04:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.509+0000 7fd678ff9640 10 client.4287.objecter ms_handle_connect 0x7fd66806c820 2023-12-01T17:04:04.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.509+0000 7fd67bfff640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd66806c820 0x7fd66806ece0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd674001c60 tx=0x7fd674007d10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.515+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd670007e40 con 0x7fd67c145d60 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.515+0000 7fd678ff9640 10 client.4287.objecter ms_dispatch 0x7fd67c000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.515+0000 7fd678ff9640 3 client.4287.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.515+0000 7fd678ff9640 3 client.4287.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 10 client.4287.objecter checking command 1 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fd668072c00 0x7fd668075020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 20 client.4287.objecter _get_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 20 client.4287.objecter put_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 15 client.4287.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 20 client.4287.objecter _get_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 20 client.4287.objecter get_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 15 client.4287.objecter _session_command_op_assign 0 1 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 20 client.4287.objecter put_session s=0x7fd668072530 osd=0 4 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 10 client.4287.objecter _send_command 1 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 --> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd6680756d0 con 0x7fd668072c00 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 20 client.4287.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd670054370 con 0x7fd67c145d60 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd678ff9640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7f04fa5d1640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0c7fd0 0x7f04ec160660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04f406a1d0 tx=0x7f04f4067960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7f04ea7fc640 10 client.?.objecter ms_handle_connect 0x7f04ec0c7fd0 2023-12-01T17:04:04.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7f04ea7fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd67b7fe640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fd668072c00 0x7fd668075020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.516+0000 7fd67b7fe640 10 monclient: get_auth_request con 0x7fd668072c00 auth_method 0 2023-12-01T17:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.518+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f766803fad0 con 0x7f76740af380 2023-12-01T17:04:04.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.519+0000 7f765bfff640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f766006ea30 0x7f7660070ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.519+0000 7f76737fe640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f766006ea30 0x7f7660070ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7f76737fe640 10 monclient: get_auth_request con 0x7f766006ea30 auth_method 0 2023-12-01T17:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== osd.2 v2:172.21.15.67:6818/2342127620 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe7540756d0 con 0x7fe754072c00 2023-12-01T17:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7fe74ffff640 10 client.4284.objecter ms_dispatch 0x7fe768000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7fe74ffff640 10 client.4284.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7fe74ffff640 20 client.4284.objecter put_session s=0x7fe754072530 osd=2 4 2023-12-01T17:04:04.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7fe74ffff640 15 client.4284.objecter _session_command_op_remove 2 1 2023-12-01T17:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7f76737fe640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f766006ea30 0x7f7660070ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7664009760 tx=0x7f7664006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.520+0000 7f765bfff640 10 client.4280.objecter ms_handle_connect 0x7f766006ea30 2023-12-01T17:04:04.523 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.522+0000 7fd678ff9640 10 client.4287.objecter ms_handle_connect 0x7fd668072c00 2023-12-01T17:04:04.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.522+0000 7fd67b7fe640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fd668072c00 0x7fd668075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f406ca70 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: dump: 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.526 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f408e070 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f406cbd0 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ea7fc640 10 monclient: dump: 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.527 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 5 monclient: authenticate success, global_id 4290 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04ebfff640 4 set_mon_vals no callback set 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 10 monclient: _renew_subs 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec034ea0 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 10 monclient: _renew_subs 2023-12-01T17:04:04.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec160d80 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.524+0000 7f04fadd2640 1 librados: init done 2023-12-01T17:04:04.529 INFO:teuthology.orchestra.run.smithi067.stdout:68719476743 2023-12-01T17:04:04.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-01T17:04:04.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.525+0000 7f04fadd2640 10 client.4290.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.525+0000 7f04fadd2640 15 client.4290.objecter _session_command_op_assign -1 1 2023-12-01T17:04:04.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.525+0000 7f04fadd2640 20 client.4290.objecter _get_session osd=-1 returning homeless 2023-12-01T17:04:04.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.525+0000 7f04fadd2640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.525+0000 7f04fadd2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.526+0000 7f04fadd2640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04ec0a3910 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7668032070 con 0x7f76740af380 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 10 client.4280.objecter ms_dispatch 0x7f7674000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 3 client.4280.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 3 client.4280.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 10 client.4280.objecter checking command 1 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f7660074e10 0x7f7660077230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 20 client.4280.objecter _get_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 20 client.4280.objecter put_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 15 client.4280.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 20 client.4280.objecter _get_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 20 client.4280.objecter get_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 15 client.4280.objecter _session_command_op_assign 4 1 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 20 client.4280.objecter put_session s=0x7f7660074740 osd=4 4 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 10 client.4280.objecter _send_command 1 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 --> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76600778e0 con 0x7f7660074e10 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 20 client.4280.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7668077aa0 con 0x7f76740af380 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.528+0000 7f765bfff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.529+0000 7f7673fff640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f7660074e10 0x7f7660077230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.529+0000 7f7673fff640 10 monclient: get_auth_request con 0x7f7660074e10 auth_method 0 2023-12-01T17:04:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.534+0000 7f765bfff640 10 client.4280.objecter ms_handle_connect 0x7f7660074e10 2023-12-01T17:04:04.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.534+0000 7f7673fff640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f7660074e10 0x7f7660077230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 1 -- 172.21.15.67:0/531762456 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc010870 con 0x7f84d00722c0 2023-12-01T17:04:04.537 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: dump: 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.538 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 1 -- 172.21.15.67:0/531762456 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc0109d0 con 0x7f84d00722c0 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 1 -- 172.21.15.67:0/531762456 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc010d80 con 0x7f84d00722c0 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84cd7fa640 10 monclient: dump: 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.539 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84ceffd640 4 set_mon_vals no callback set 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 5 monclient: authenticate success, global_id 4286 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 10 monclient: _renew_subs 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d01384f0 con 0x7f84d00722c0 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 10 monclient: _renew_subs 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d013ab50 con 0x7f84d00722c0 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7f84d6df0640 1 librados: init done 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== osd.7 v2:172.21.15.92:6808/2695530702 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fea18076160 con 0x7fea180735f0 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7fea2ffff640 10 client.4274.objecter ms_dispatch 0x7fea38000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7fea2ffff640 10 client.4274.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7fea2ffff640 20 client.4274.objecter put_session s=0x7fea18072f20 osd=7 4 2023-12-01T17:04:04.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.536+0000 7fea2ffff640 15 client.4274.objecter _session_command_op_remove 7 1 2023-12-01T17:04:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.538+0000 7f84d6df0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.538+0000 7f84d6df0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.538+0000 7f84d6df0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.538+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d0138ec0 con 0x7f84d00722c0 2023-12-01T17:04:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 10 client.4284.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 15 client.4284.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 20 client.4284.objecter _get_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.552 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 20 client.4284.objecter put_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 20 client.4284.objecter _get_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 15 client.4284.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 20 client.4284.objecter get_session s=0x7fe754072530 osd=2 3 2023-12-01T17:04:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 15 client.4284.objecter _session_command_op_assign 2 2 2023-12-01T17:04:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 20 client.4284.objecter put_session s=0x7fe754072530 osd=2 4 2023-12-01T17:04:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 10 client.4284.objecter _send_command 2 2023-12-01T17:04:04.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.551+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe76814fec0 con 0x7fe754072c00 2023-12-01T17:04:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.557+0000 7fe74ffff640 1 -- 172.21.15.67:0/1504842160 <== osd.2 v2:172.21.15.67:6818/2342127620 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe76814fec0 con 0x7fe754072c00 2023-12-01T17:04:04.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.557+0000 7fe74ffff640 10 client.4284.objecter ms_dispatch 0x7fe768000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.559 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.557+0000 7fe74ffff640 10 client.4284.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.559 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.557+0000 7fe74ffff640 20 client.4284.objecter put_session s=0x7fe754072530 osd=2 4 2023-12-01T17:04:04.559 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.557+0000 7fe74ffff640 15 client.4284.objecter _session_command_op_remove 2 2 2023-12-01T17:04:04.560 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.557+0000 7fe76dcc2640 10 librados: watch_flush enter 2023-12-01T17:04:04.560 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 10 librados: watch_flush exit 2023-12-01T17:04:04.560 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 10 client.4284.objecter close_session for osd.2 2023-12-01T17:04:04.560 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7fe754072c00 msgr2=0x7fe754075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7fe754072c00 0x7fe754075020 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 20 client.4284.objecter put_session s=0x7fe754072530 osd=2 1 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 20 client.4284.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 10 client.4284.objecter successfully canceled tick 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe75406c820 msgr2=0x7fe75406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe75406c820 0x7fe75406ece0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe758009710 tx=0x7fe758006c90 comp rx=0 tx=0).stop 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 10 monclient: shutdown 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe768152980 msgr2=0x7fe768150170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.558+0000 7fe76dcc2640 1 --2- 172.21.15.67:0/1504842160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe768152980 0x7fe768150170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00b4c0 tx=0x7fe75c00b980 comp rx=0 tx=0).stop 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.559+0000 7fe767fff640 1 -- 172.21.15.67:0/1504842160 reap_dead start 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.559+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 shutdown_connections 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.559+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 >> 172.21.15.67:0/1504842160 conn(0x7fe7680b3b20 msgr2=0x7fe7680b22a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.559+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 shutdown_connections 2023-12-01T17:04:04.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.559+0000 7fe76dcc2640 1 -- 172.21.15.67:0/1504842160 wait complete. 2023-12-01T17:04:04.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.559+0000 7fe76dcc2640 1 librados: shutdown 2023-12-01T17:04:04.569 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.567+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f04f406cd30 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.569 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.568+0000 7f04ea7fc640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f04d806c870 0x7f04d806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.568+0000 7f04f9dd0640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f04d806c870 0x7f04d806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.569+0000 7f04f9dd0640 10 monclient: get_auth_request con 0x7f04d806c870 auth_method 0 2023-12-01T17:04:04.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.569+0000 7f04ea7fc640 10 client.4290.objecter ms_handle_connect 0x7f04d806c870 2023-12-01T17:04:04.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.569+0000 7f04f9dd0640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f04d806c870 0x7f04d806ed30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f04ec15aa10 tx=0x7f04f0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 10 client.4274.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 15 client.4274.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 20 client.4274.objecter _get_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 20 client.4274.objecter put_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 20 client.4274.objecter _get_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 15 client.4274.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 20 client.4274.objecter get_session s=0x7fea18072f20 osd=7 3 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 15 client.4274.objecter _session_command_op_assign 7 2 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 20 client.4274.objecter put_session s=0x7fea18072f20 osd=7 4 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 10 client.4274.objecter _send_command 2 2023-12-01T17:04:04.572 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7fea3d3ce640 1 -- 172.21.15.67:0/3519114349 --> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fea38149400 con 0x7fea180735f0 2023-12-01T17:04:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.571+0000 7f84cd7fa640 1 -- 172.21.15.67:0/531762456 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f84bc023480 con 0x7f84d00722c0 2023-12-01T17:04:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.572+0000 7f84cd7fa640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f84b406c870 0x7f84b406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.572+0000 7f84d4b8c640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f84b406c870 0x7f84b406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.573 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.572+0000 7f84d4b8c640 10 monclient: get_auth_request con 0x7f84b406c870 auth_method 0 2023-12-01T17:04:04.574 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.573+0000 7f84d4b8c640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f84b406c870 0x7f84b406ed30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f84c4001560 tx=0x7f84c4007990 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.574 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.573+0000 7f84cd7fa640 10 client.4286.objecter ms_handle_connect 0x7f84b406c870 2023-12-01T17:04:04.576 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f04f40d28a0 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 10 client.4290.objecter ms_dispatch 0x7f04f4000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 3 client.4290.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 3 client.4290.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 10 client.4290.objecter checking command 1 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 0x7f04d8075070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 20 client.4290.objecter _get_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 20 client.4290.objecter put_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 15 client.4290.objecter _session_command_op_remove -1 1 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 20 client.4290.objecter _get_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 20 client.4290.objecter get_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 15 client.4290.objecter _session_command_op_assign 1 1 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 20 client.4290.objecter put_session s=0x7f04d8072580 osd=1 4 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 10 client.4290.objecter _send_command 1 2023-12-01T17:04:04.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04d8075720 con 0x7f04d8072c50 2023-12-01T17:04:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 20 client.4290.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04f40ce4e0 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.578 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.575+0000 7f04ea7fc640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:04:04.578 INFO:teuthology.orchestra.run.smithi067.stdout:68719476742 2023-12-01T17:04:04.579 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-01T17:04:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2ffff640 1 -- 172.21.15.67:0/3519114349 <== osd.7 v2:172.21.15.92:6808/2695530702 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fea38149400 con 0x7fea180735f0 2023-12-01T17:04:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2ffff640 10 client.4274.objecter ms_dispatch 0x7fea38000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2ffff640 10 client.4274.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2ffff640 20 client.4274.objecter put_session s=0x7fea18072f20 osd=7 4 2023-12-01T17:04:04.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2ffff640 15 client.4274.objecter _session_command_op_remove 7 2 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 10 librados: watch_flush enter 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 10 librados: watch_flush exit 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 10 client.4274.objecter close_session for osd.7 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 -- 172.21.15.67:0/3519114349 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fea180735f0 msgr2=0x7fea18075a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fea180735f0 0x7fea18075a90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 20 client.4274.objecter put_session s=0x7fea18072f20 osd=7 1 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 20 client.4274.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 10 client.4274.objecter successfully canceled tick 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 -- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fea1806c870 msgr2=0x7fea1806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.583 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fea1806c870 0x7fea1806ed30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fea20035000 tx=0x7fea20030000 comp rx=0 tx=0).stop 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 10 monclient: shutdown 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 -- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea38061190 msgr2=0x7fea38147860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 --2- 172.21.15.67:0/3519114349 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fea38061190 0x7fea38147860 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c430 tx=0x7fea2800c8f0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea377fe640 1 -- 172.21.15.67:0/3519114349 reap_dead start 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 -- 172.21.15.67:0/3519114349 shutdown_connections 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.578+0000 7fea2d7fa640 1 -- 172.21.15.67:0/3519114349 >> 172.21.15.67:0/3519114349 conn(0x7fea3806a500 msgr2=0x7fea38140220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.579+0000 7fea2d7fa640 1 -- 172.21.15.67:0/3519114349 shutdown_connections 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.579+0000 7fea2d7fa640 1 -- 172.21.15.67:0/3519114349 wait complete. 2023-12-01T17:04:04.584 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.579+0000 7fea2d7fa640 1 librados: shutdown 2023-12-01T17:04:04.589 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.588+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== osd.0 v2:172.21.15.67:6810/358761697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd6680756d0 con 0x7fd668072c00 2023-12-01T17:04:04.589 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.588+0000 7fd678ff9640 10 client.4287.objecter ms_dispatch 0x7fd67c000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.589 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.588+0000 7fd678ff9640 10 client.4287.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.589 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.588+0000 7fd678ff9640 20 client.4287.objecter put_session s=0x7fd668072530 osd=0 4 2023-12-01T17:04:04.589 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.588+0000 7fd678ff9640 15 client.4287.objecter _session_command_op_remove 0 1 2023-12-01T17:04:04.592 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2023-12-01T17:04:04.592 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-01T17:04:04.594 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.593+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== osd.4 v2:172.21.15.92:6816/2070368587 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f76600778e0 con 0x7f7660074e10 2023-12-01T17:04:04.594 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.593+0000 7f765bfff640 10 client.4280.objecter ms_dispatch 0x7f7674000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.594 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.593+0000 7f765bfff640 10 client.4280.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.594 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.593+0000 7f765bfff640 20 client.4280.objecter put_session s=0x7f7660074740 osd=4 4 2023-12-01T17:04:04.594 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.593+0000 7f765bfff640 15 client.4280.objecter _session_command_op_remove 4 1 2023-12-01T17:04:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.611+0000 7fe7e6ffd640 1 -- 172.21.15.67:0/4065097718 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe7ec013470 con 0x7fe7f81434a0 2023-12-01T17:04:04.613 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.611+0000 7fe7e6ffd640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d806c820 0x7fe7d806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.612+0000 7fe7f659c640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d806c820 0x7fe7d806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.612+0000 7fe7f659c640 10 monclient: get_auth_request con 0x7fe7d806c820 auth_method 0 2023-12-01T17:04:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.613+0000 7fe7e6ffd640 1 -- 172.21.15.67:0/4065097718 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe7ec058a40 con 0x7fe7f81434a0 2023-12-01T17:04:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.613+0000 7fe7e6ffd640 10 client.4330.objecter ms_dispatch 0x7fe7f8000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.613+0000 7fe7e6ffd640 3 client.4330.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.614 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.613+0000 7fe7e6ffd640 3 client.4330.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.613+0000 7fe7e6ffd640 20 client.4330.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.613+0000 7fe7f659c640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d806c820 0x7fe7d806ece0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8000ff0 tx=0x7fe7e80014d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.615 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.613+0000 7fe7e6ffd640 10 client.4330.objecter ms_handle_connect 0x7fe7d806c820 2023-12-01T17:04:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.617+0000 7fd6817b3640 10 client.4287.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.617+0000 7fd6817b3640 15 client.4287.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 20 client.4287.objecter _get_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 20 client.4287.objecter put_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 20 client.4287.objecter _get_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 15 client.4287.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 20 client.4287.objecter get_session s=0x7fd668072530 osd=0 3 2023-12-01T17:04:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 15 client.4287.objecter _session_command_op_assign 0 2 2023-12-01T17:04:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 20 client.4287.objecter put_session s=0x7fd668072530 osd=0 4 2023-12-01T17:04:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 10 client.4287.objecter _send_command 2 2023-12-01T17:04:04.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.618+0000 7fd6817b3640 1 -- 172.21.15.67:0/874797375 --> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd67c0b0220 con 0x7fd668072c00 2023-12-01T17:04:04.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.622+0000 7fe7e6ffd640 1 -- 172.21.15.67:0/4065097718 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7ec054d80 con 0x7fe7f81434a0 2023-12-01T17:04:04.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.622+0000 7fe7e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.624 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.622+0000 7fe7e6ffd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:04.626 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 15 client.4280.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 20 client.4280.objecter _get_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 20 client.4280.objecter put_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 20 client.4280.objecter _get_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 15 client.4280.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 20 client.4280.objecter get_session s=0x7f7660074740 osd=4 3 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 15 client.4280.objecter _session_command_op_assign 4 2 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 20 client.4280.objecter put_session s=0x7f7660074740 osd=4 4 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 10 client.4280.objecter _send_command 2 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 --> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f76740afd80 con 0x7f7660074e10 2023-12-01T17:04:04.627 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f04f95cf640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 0x7f04d8075070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.628 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.624+0000 7f04f95cf640 10 monclient: get_auth_request con 0x7f04d8072c50 auth_method 0 2023-12-01T17:04:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.634+0000 7fd678ff9640 1 -- 172.21.15.67:0/874797375 <== osd.0 v2:172.21.15.67:6810/358761697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd67c0b0220 con 0x7fd668072c00 2023-12-01T17:04:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.634+0000 7fd678ff9640 10 client.4287.objecter ms_dispatch 0x7fd67c000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd678ff9640 10 client.4287.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd678ff9640 20 client.4287.objecter put_session s=0x7fd668072530 osd=0 4 2023-12-01T17:04:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7f04ea7fc640 10 client.4290.objecter ms_handle_connect 0x7f04d8072c50 2023-12-01T17:04:04.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7f04f95cf640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 0x7f04d8075070 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.637 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd678ff9640 15 client.4287.objecter _session_command_op_remove 0 2 2023-12-01T17:04:04.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 10 librados: watch_flush enter 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 10 librados: watch_flush exit 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 10 client.4287.objecter close_session for osd.0 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 1 -- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fd668072c00 msgr2=0x7fd668075020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fd668072c00 0x7fd668075020 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 20 client.4287.objecter put_session s=0x7fd668072530 osd=0 1 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 20 client.4287.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.635+0000 7fd6627fc640 10 client.4287.objecter successfully canceled tick 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 1 -- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd66806c820 msgr2=0x7fd66806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd66806c820 0x7fd66806ece0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd674001c60 tx=0x7fd674007d10 comp rx=0 tx=0).stop 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 10 monclient: shutdown 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 1 -- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c145d60 msgr2=0x7fd67c144490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 1 --2- 172.21.15.67:0/874797375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd67c145d60 0x7fd67c144490 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd67000c3c0 tx=0x7fd670001e70 comp rx=0 tx=0).stop 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd680fb2640 1 -- 172.21.15.67:0/874797375 reap_dead start 2023-12-01T17:04:04.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 1 -- 172.21.15.67:0/874797375 shutdown_connections 2023-12-01T17:04:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.636+0000 7fd6627fc640 1 -- 172.21.15.67:0/874797375 >> 172.21.15.67:0/874797375 conn(0x7fd67c06c0c0 msgr2=0x7fd67c0b2cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.638+0000 7fd6627fc640 1 -- 172.21.15.67:0/874797375 shutdown_connections 2023-12-01T17:04:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.638+0000 7fd6627fc640 1 -- 172.21.15.67:0/874797375 wait complete. 2023-12-01T17:04:04.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.638+0000 7fd6627fc640 1 librados: shutdown 2023-12-01T17:04:04.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.652+0000 7f765bfff640 1 -- 172.21.15.67:0/3391583898 <== osd.4 v2:172.21.15.92:6816/2070368587 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f76740afd80 con 0x7f7660074e10 2023-12-01T17:04:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.652+0000 7f765bfff640 10 client.4280.objecter ms_dispatch 0x7f7674000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.652+0000 7f765bfff640 10 client.4280.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.652+0000 7f765bfff640 20 client.4280.objecter put_session s=0x7f7660074740 osd=4 4 2023-12-01T17:04:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.652+0000 7f765bfff640 15 client.4280.objecter _session_command_op_remove 4 2 2023-12-01T17:04:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 10 librados: watch_flush enter 2023-12-01T17:04:04.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 10 librados: watch_flush exit 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 10 client.4280.objecter close_session for osd.4 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f7660074e10 msgr2=0x7f7660077230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f7660074e10 0x7f7660077230 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 20 client.4280.objecter put_session s=0x7f7660074740 osd=4 1 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 20 client.4280.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 10 client.4280.objecter successfully canceled tick 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f766006ea30 msgr2=0x7f7660070ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f766006ea30 0x7f7660070ef0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7664009760 tx=0x7f7664006c90 comp rx=0 tx=0).stop 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 10 monclient: shutdown 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f76740af380 msgr2=0x7f76741d6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 --2- 172.21.15.67:0/3391583898 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f76740af380 0x7f76741d6920 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f76680090b0 tx=0x7f76680046c0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f7673fff640 1 -- 172.21.15.67:0/3391583898 reap_dead start 2023-12-01T17:04:04.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 shutdown_connections 2023-12-01T17:04:04.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.653+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 >> 172.21.15.67:0/3391583898 conn(0x7f76740b3ef0 msgr2=0x7f76741c9750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.654+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 shutdown_connections 2023-12-01T17:04:04.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.654+0000 7f767a02b640 1 -- 172.21.15.67:0/3391583898 wait complete. 2023-12-01T17:04:04.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.654+0000 7f767a02b640 1 librados: shutdown 2023-12-01T17:04:04.657 INFO:teuthology.orchestra.run.smithi067.stdout:64424509444 2023-12-01T17:04:04.657 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-01T17:04:04.672 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2023-12-01T17:04:04.672 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-01T17:04:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.693+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== osd.1 v2:172.21.15.67:6826/866081726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f04d8075720 con 0x7f04d8072c50 2023-12-01T17:04:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.693+0000 7f04ea7fc640 10 client.4290.objecter ms_dispatch 0x7f04f4000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:04:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.693+0000 7f04ea7fc640 10 client.4290.objecter _finish_command 1 = system:0 2023-12-01T17:04:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.694+0000 7f04ea7fc640 20 client.4290.objecter put_session s=0x7f04d8072580 osd=1 4 2023-12-01T17:04:04.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.694+0000 7f04ea7fc640 15 client.4290.objecter _session_command_op_remove 1 1 2023-12-01T17:04:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.722+0000 7fa6ab37d640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.722+0000 7fa6ab37d640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.722+0000 7fa6ab37d640 10 monclient: monmap: 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.723994+0000 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.723994+0000 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.727 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.728 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.728 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.728 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.730+0000 7fa6ab37d640 1 Processor -- start 2023-12-01T17:04:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 10 client.4290.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-12-01T17:04:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 15 client.4290.objecter _session_command_op_assign -1 2 2023-12-01T17:04:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 20 client.4290.objecter _get_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 20 client.4290.objecter put_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 20 client.4290.objecter _get_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 15 client.4290.objecter _session_command_op_remove -1 2 2023-12-01T17:04:04.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 20 client.4290.objecter get_session s=0x7f04d8072580 osd=1 3 2023-12-01T17:04:04.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 15 client.4290.objecter _session_command_op_assign 1 2 2023-12-01T17:04:04.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 20 client.4290.objecter put_session s=0x7f04d8072580 osd=1 4 2023-12-01T17:04:04.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 10 client.4290.objecter _send_command 2 2023-12-01T17:04:04.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7f04fadd2640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f04ec007750 con 0x7f04d8072c50 2023-12-01T17:04:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7fa6ab37d640 1 -- start start 2023-12-01T17:04:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.732+0000 7fa6ab37d640 10 monclient: init 2023-12-01T17:04:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-12-01T17:04:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa6a413ec40 0x7fa6a40aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6a413ec40 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:04.735 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a40ae100 0x7fa6a413b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6a40ae100 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa6a406f800 0x7fa6a4071f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6a406f800 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.736 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.743 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a413df50 con 0x7fa6a406f800 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a4070600 con 0x7fa6a413ec40 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a40af020 con 0x7fa6a40ae100 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.733+0000 7fa6ab37d640 10 monclient(hunting): authenticate will time out at 1375.971795s 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a9119640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa6a413ec40 0x7fa6a40aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a9119640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa6a413ec40 0x7fa6a40aeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36594/0 (socket says 172.21.15.67:36594) 2023-12-01T17:04:04.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a9119640 1 -- 172.21.15.67:0/2008272185 learned_addr learned my addr 172.21.15.67:0/2008272185 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a9119640 10 monclient(hunting): get_auth_request con 0x7fa6a413ec40 auth_method 0 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a9119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a9119640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a9119640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a991a640 1 --2- 172.21.15.67:0/2008272185 >> v2:172.21.15.67:3300/0 conn(0x7fa6a406f800 0x7fa6a4071f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a991a640 10 monclient(hunting): get_auth_request con 0x7fa6a406f800 auth_method 0 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a991a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a991a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.735+0000 7fa6a991a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.737+0000 7fa6a9119640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.737+0000 7fa6a9119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.737+0000 7fa6a9119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.737+0000 7fa6a991a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.737+0000 7fa6a991a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.737+0000 7fa6a991a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 10 monclient(hunting): handle_auth_done global_id 4333 payload 274 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 1 -- 172.21.15.67:0/2008272185 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a40ae100 msgr2=0x7fa6a413b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 1 --2- 172.21.15.67:0/2008272185 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a40ae100 0x7fa6a413b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 1 -- 172.21.15.67:0/2008272185 >> v2:172.21.15.67:3300/0 conn(0x7fa6a406f800 msgr2=0x7fa6a4071f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 1 --2- 172.21.15.67:0/2008272185 >> v2:172.21.15.67:3300/0 conn(0x7fa6a406f800 0x7fa6a4071f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.741+0000 7fa6a9119640 1 -- 172.21.15.67:0/2008272185 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a413ba20 con 0x7fa6a413ec40 2023-12-01T17:04:04.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.742+0000 7fa6a991a640 1 --2- 172.21.15.67:0/2008272185 >> v2:172.21.15.67:3300/0 conn(0x7fa6a406f800 0x7fa6a4071f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-01T17:04:04.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.746+0000 7fa6a9119640 1 --2- 172.21.15.67:0/2008272185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a413ec40 0x7fa6a40aeae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40b1270 tx=0x7fa69402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34f53271c7b1a493 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.752 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 1 -- 172.21.15.67:0/2008272185 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6940317e0 con 0x7fa6a413ec40 2023-12-01T17:04:04.752 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.752 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: dump: 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.753 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 1 -- 172.21.15.67:0/2008272185 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa694031940 con 0x7fa6a413ec40 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 1 -- 172.21.15.67:0/2008272185 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694031cd0 con 0x7fa6a413ec40 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.750+0000 7fa69bfff640 10 monclient: dump: 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.754 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.755 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.755 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.751+0000 7fa6ab37d640 5 monclient: authenticate success, global_id 4333 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.751+0000 7fa6ab37d640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.751+0000 7fa6ab37d640 4 set_mon_vals no callback set 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.752+0000 7fa6ab37d640 10 monclient: shutdown 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.752+0000 7fa6ab37d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.752+0000 7fa6ab37d640 1 -- 172.21.15.67:0/2008272185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a413ec40 msgr2=0x7fa6a40aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.752+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/2008272185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a413ec40 0x7fa6a40aeae0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa6a40b1270 tx=0x7fa69402b880 comp rx=0 tx=0).stop 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.753+0000 7fa6ab37d640 1 -- 172.21.15.67:0/2008272185 shutdown_connections 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.753+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/2008272185 >> v2:172.21.15.67:3300/0 conn(0x7fa6a406f800 0x7fa6a4071f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.753+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/2008272185 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a40ae100 0x7fa6a413b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.753+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/2008272185 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a413ec40 0x7fa6a40aeae0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.753+0000 7fa6ab37d640 1 -- 172.21.15.67:0/2008272185 >> 172.21.15.67:0/2008272185 conn(0x7fa6a40b3b20 msgr2=0x7fa6a40b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.753+0000 7fa6ab37d640 1 -- 172.21.15.67:0/2008272185 shutdown_connections 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.753+0000 7fa6ab37d640 1 -- 172.21.15.67:0/2008272185 wait complete. 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.754+0000 7fa6ab37d640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.754+0000 7fa6ab37d640 10 monclient: monmap: 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.755+0000 7fa6ab37d640 1 librados: starting msgr at 2023-12-01T17:04:04.758 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.755+0000 7fa6ab37d640 1 librados: starting objecter 2023-12-01T17:04:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.755+0000 7fa6ab37d640 1 Processor -- start 2023-12-01T17:04:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.755+0000 7fa6ab37d640 1 -- start start 2023-12-01T17:04:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.755+0000 7fa6ab37d640 1 librados: setting wanted keys 2023-12-01T17:04:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.755+0000 7fa6ab37d640 1 librados: calling monclient init 2023-12-01T17:04:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.755+0000 7fa6ab37d640 10 monclient: init 2023-12-01T17:04:04.759 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.756+0000 7f06a3bf0640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.756+0000 7f06a3bf0640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.756+0000 7f06a3bf0640 10 monclient: monmap: 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.758202+0000 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.758202+0000 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.760 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.756+0000 7fa6ab37d640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.756+0000 7fa6ab37d640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.756+0000 7fa6ab37d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a406f800 0x7fa6a4141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.757+0000 7fa6ab37d640 10 monclient(hunting): picked mon.noname-a con 0x7fa6a406f800 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.757+0000 7fa6ab37d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a40ae100 0x7fa6a412e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.757+0000 7fa6ab37d640 10 monclient(hunting): picked mon.noname-b con 0x7fa6a40ae100 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.757+0000 7fa6ab37d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a413ec40 0x7fa6a41299b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.757+0000 7fa6ab37d640 10 monclient(hunting): picked mon.noname-c con 0x7fa6a413ec40 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6a9119640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a406f800 0x7fa6a4141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a40a8a20 con 0x7fa6a406f800 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6a8918640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a40ae100 0x7fa6a412e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a412f000 con 0x7fa6a40ae100 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6a9119640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a406f800 0x7fa6a4141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57972/0 (socket says 172.21.15.67:57972) 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6a9119640 1 -- 172.21.15.67:0/1865339612 learned_addr learned my addr 172.21.15.67:0/1865339612 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a412f520 con 0x7fa6a413ec40 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.758+0000 7fa6ab37d640 10 monclient(hunting): authenticate will time out at 1375.997378s 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a9119640 10 monclient(hunting): get_auth_request con 0x7fa6a406f800 auth_method 0 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a9119640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a9119640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a9119640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a8918640 10 monclient(hunting): get_auth_request con 0x7fa6a40ae100 auth_method 0 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a8918640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a8918640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.759+0000 7fa6a8918640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fa6a8918640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fa6a8918640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fa6a8918640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fa6a9119640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fa6a9119640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fa6a9119640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.762+0000 7fa6a8918640 10 monclient(hunting): handle_auth_done global_id 4336 payload 824 2023-12-01T17:04:04.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 1 -- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a413ec40 msgr2=0x7fa6a41299b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a413ec40 0x7fa6a41299b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 1 -- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a406f800 msgr2=0x7fa6a4141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a406f800 0x7fa6a4141090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fc03d239640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fc03d239640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.760+0000 7fc03d239640 10 monclient: monmap: 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.762349+0000 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.762349+0000 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.761+0000 7fc03d239640 1 Processor -- start 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.761+0000 7fc03d239640 1 -- start start 2023-12-01T17:04:04.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.761+0000 7fc03d239640 10 monclient: init 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.762+0000 7fc03d239640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.762+0000 7fc03d239640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc030006e80 0x7fc0300e2890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 10 monclient(hunting): picked mon.noname-c con 0x7fc030006e80 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc030030d40 0x7fc0300e2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 10 monclient(hunting): picked mon.noname-a con 0x7fc030030d40 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc030032180 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc030033200 con 0x7fc030032180 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc030007260 con 0x7fc030030d40 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0300312d0 con 0x7fc030006e80 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fc03d239640 10 monclient(hunting): authenticate will time out at 1376.001499s 2023-12-01T17:04:04.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7fa6a8918640 1 -- 172.21.15.67:0/1865339612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a0009400 con 0x7fa6a40ae100 2023-12-01T17:04:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7f06a3bf0640 1 Processor -- start 2023-12-01T17:04:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7f06a3bf0640 1 -- start start 2023-12-01T17:04:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.763+0000 7f06a3bf0640 10 monclient: init 2023-12-01T17:04:04.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c0a8a20 0x7f069c0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient(hunting): picked mon.noname-b con 0x7f069c0a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f069c145ab0 0x7f069c145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient(hunting): picked mon.noname-c con 0x7f069c145ab0 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f069c13f0f0 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f069c146430 con 0x7f069c13f0f0 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f069c145620 con 0x7f069c145ab0 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c0aace0 con 0x7f069c0a8a20 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.772 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7f06a3bf0640 10 monclient(hunting): authenticate will time out at 1376.003387s 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.764+0000 7fc0377fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc030030d40 0x7fc0300e2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc037fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc030006e80 0x7fc0300e2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc037fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc030006e80 0x7fc0300e2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36616/0 (socket says 172.21.15.67:36616) 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc037fff640 1 -- 172.21.15.67:0/3831752451 learned_addr learned my addr 172.21.15.67:0/3831752451 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc037fff640 10 monclient(hunting): get_auth_request con 0x7fc030006e80 auth_method 0 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc037fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc037fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.773 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc037fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc0377fe640 10 monclient(hunting): get_auth_request con 0x7fc030030d40 auth_method 0 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc0377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc0377fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.765+0000 7fc0377fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.766+0000 7fc037fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.766+0000 7fc037fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.766+0000 7fc037fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.766+0000 7f06a118b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f069c145ab0 0x7f069c145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.775 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.775 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc0377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 10 monclient(hunting): handle_auth_done global_id 4339 payload 274 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 1 -- 172.21.15.67:0/3831752451 >> v2:172.21.15.67:3300/0 conn(0x7fc030030d40 msgr2=0x7fc0300e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 1 --2- 172.21.15.67:0/3831752451 >> v2:172.21.15.67:3300/0 conn(0x7fc030030d40 0x7fc0300e2dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 1 -- 172.21.15.67:0/3831752451 >> v1:172.21.15.67:6789/0 conn(0x7fc030032180 legacy=0x7fc0300e8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.776 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.768+0000 7fc037fff640 1 -- 172.21.15.67:0/3831752451 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300e3310 con 0x7fc030006e80 2023-12-01T17:04:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.769+0000 7f06a218d640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f069c13f0f0 0x7f069c13f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57798/0 (socket says 172.21.15.67:57798) 2023-12-01T17:04:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.769+0000 7f06a218d640 1 -- 172.21.15.67:0/2341011122 learned_addr learned my addr 172.21.15.67:0/2341011122 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.769+0000 7f06a118b640 10 monclient(hunting): get_auth_request con 0x7f069c145ab0 auth_method 0 2023-12-01T17:04:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.769+0000 7f06a118b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.769+0000 7f06a118b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.777 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.769+0000 7f06a118b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.771+0000 7f06a118b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.771+0000 7f06a118b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.771+0000 7f06a118b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.778 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.771+0000 7fc037fff640 1 --2- 172.21.15.67:0/3831752451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030006e80 0x7fc0300e2890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc030030a90 tx=0x7fc02c0300a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=27a9af5ecbcab8cc server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.772+0000 7f06a118b640 10 monclient(hunting): handle_auth_done global_id 4342 payload 274 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.772+0000 7f06a118b640 1 -- 172.21.15.67:0/2341011122 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c0a8a20 msgr2=0x7f069c0a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.772+0000 7f06a118b640 1 --2- 172.21.15.67:0/2341011122 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c0a8a20 0x7f069c0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.773+0000 7f06a118b640 1 -- 172.21.15.67:0/2341011122 >> v1:172.21.15.92:6789/0 conn(0x7f069c13f0f0 legacy=0x7f069c13f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.773+0000 7f06a118b640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.773+0000 7f06a118b640 1 monclient: found mon.noname-c 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.773+0000 7f06a118b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.773+0000 7f06a118b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.773+0000 7f06a118b640 1 -- 172.21.15.67:0/2341011122 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c0aafc0 con 0x7f069c145ab0 2023-12-01T17:04:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.774+0000 7fa6a8918640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a40ae100 0x7fa6a412e7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa69c00bd70 tx=0x7fa69c00be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.774+0000 7fa699ffb640 10 client.?.objecter ms_handle_connect 0x7fa6a40ae100 2023-12-01T17:04:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.774+0000 7fa699ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.780 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.776+0000 7f06a118b640 1 --2- 172.21.15.67:0/2341011122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c145ab0 0x7f069c145ef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f06980097a0 tx=0x7f069802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c19731a25433a3b server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.779+0000 7fc036ffd640 1 -- 172.21.15.67:0/3831752451 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c030e50 con 0x7fc030006e80 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.779+0000 7fc036ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.779+0000 7fc036ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.779+0000 7fc036ffd640 10 monclient: dump: 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.781 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.779+0000 7fc036ffd640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.779+0000 7fc036ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7fc036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7fc03d239640 5 monclient: authenticate success, global_id 4339 2023-12-01T17:04:04.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7fc03d239640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a098a640 1 -- 172.21.15.67:0/2341011122 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698031780 con 0x7f069c145ab0 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a098a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a098a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a098a640 10 monclient: dump: 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.783 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a098a640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a098a640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a098a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a3bf0640 5 monclient: authenticate success, global_id 4342 2023-12-01T17:04:04.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.780+0000 7f06a3bf0640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04f95cf640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 msgr2=0x7f04d8075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04f95cf640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 msgr2=0x7f04d8075070 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04f95cf640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 0x7f04d8075070 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04f95cf640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 0x7f04d8075070 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 1 client.4290.objecter ms_handle_reset 0x7f04d8072c50 session 0x7f04d8072580 osd.1 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 10 client.4290.objecter reopen_session osd.1 session, addr now [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 msgr2=0x7f04d8075070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8072c50 0x7f04d8075070 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8075a10 0x7f04d8077e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 10 client.4290.objecter _send_command 2 2023-12-01T17:04:04.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f04d8075720 con 0x7f04d8075a10 2023-12-01T17:04:04.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 10 monclient: _renew_subs 2023-12-01T17:04:04.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.781+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7f04d80783e0 con 0x7f04ec0c7fd0 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa699ffb640 1 -- 172.21.15.67:0/1865339612 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c014440 con 0x7fa6a40ae100 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa699ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa699ffb640 10 monclient: dump: 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.788 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa699ffb640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa699ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa6ab37d640 5 monclient: authenticate success, global_id 4336 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa6ab37d640 10 monclient: _renew_subs 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa6ab37d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.782+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6940094d0 con 0x7fa6a40ae100 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fa6ab37d640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fa6ab37d640 10 monclient: _renew_subs 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fa6ab37d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a412df30 con 0x7fa6a40ae100 2023-12-01T17:04:04.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fa6ab37d640 1 librados: init done 2023-12-01T17:04:04.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fc036ffd640 1 -- 172.21.15.67:0/3831752451 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c034070 con 0x7fc030006e80 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fc036ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fc03d239640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.783+0000 7fc03d239640 4 set_mon_vals no callback set 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 10 monclient: shutdown 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 -- 172.21.15.67:0/3831752451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030006e80 msgr2=0x7fc0300e2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 --2- 172.21.15.67:0/3831752451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030006e80 0x7fc0300e2890 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc030030a90 tx=0x7fc02c0300a0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 -- 172.21.15.67:0/3831752451 shutdown_connections 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 --2- 172.21.15.67:0/3831752451 >> v2:172.21.15.67:3300/0 conn(0x7fc030030d40 0x7fc0300e2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 --2- 172.21.15.67:0/3831752451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030006e80 0x7fc0300e2890 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 -- 172.21.15.67:0/3831752451 >> 172.21.15.67:0/3831752451 conn(0x7fc030005ee0 msgr2=0x7fc0300062f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 -- 172.21.15.67:0/3831752451 shutdown_connections 2023-12-01T17:04:04.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fc03d239640 1 -- 172.21.15.67:0/3831752451 wait complete. 2023-12-01T17:04:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fa6ab37d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fa6ab37d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fa6ab37d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.784+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a404e840 con 0x7fa6a40ae100 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 10 monclient: monmap: 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.793 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 1 librados: starting msgr at 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 1 librados: starting objecter 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 1 Processor -- start 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 1 -- start start 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 1 librados: setting wanted keys 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 1 librados: calling monclient init 2023-12-01T17:04:04.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.785+0000 7fc03d239640 10 monclient: init 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fa699ffb640 1 -- 172.21.15.67:0/1865339612 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c009870 con 0x7fa6a40ae100 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fa699ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fa699ffb640 1 -- 172.21.15.67:0/1865339612 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c0147c0 con 0x7fa6a40ae100 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fa699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fa699ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fa699ffb640 10 monclient: dump: 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.795 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fc03d239640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.796 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fa69b7fe640 4 set_mon_vals no callback set 2023-12-01T17:04:04.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fc03d239640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:04.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fc03d239640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc030006e80 0x7fc030186c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fc03d239640 10 monclient(hunting): picked mon.noname-c con 0x7fc030006e80 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fc03d239640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030030d40 0x7fc0300d94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fc03d239640 10 monclient(hunting): picked mon.noname-b con 0x7fc030030d40 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.786+0000 7fc03d239640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc030032180 0x7fc030187330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 10 monclient(hunting): picked mon.noname-a con 0x7fc030032180 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300d8b60 con 0x7fc030032180 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc030187870 con 0x7fc030030d40 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300d8e00 con 0x7fc030006e80 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03d239640 10 monclient(hunting): authenticate will time out at 1376.025518s 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.787+0000 7fc03ca38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc030032180 0x7fc030187330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.788+0000 7fc03ca38640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc030032180 0x7fc030187330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58002/0 (socket says 172.21.15.67:58002) 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.788+0000 7fc03ca38640 1 -- 172.21.15.67:0/43435033 learned_addr learned my addr 172.21.15.67:0/43435033 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.788+0000 7fc03ca38640 10 monclient(hunting): get_auth_request con 0x7fc030032180 auth_method 0 2023-12-01T17:04:04.799 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.788+0000 7fc03ca38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.788+0000 7fc03ca38640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.788+0000 7fc03ca38640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a098a640 1 -- 172.21.15.67:0/2341011122 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06980318e0 con 0x7f069c145ab0 2023-12-01T17:04:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a098a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7fc03ca38640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7fc03ca38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7fc03ca38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a3bf0640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a3bf0640 4 set_mon_vals no callback set 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a3bf0640 10 monclient: shutdown 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a3bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a098a640 1 -- 172.21.15.67:0/2341011122 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0698031c70 con 0x7f069c145ab0 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a3bf0640 1 -- 172.21.15.67:0/2341011122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c145ab0 msgr2=0x7f069c145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.789+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/2341011122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c145ab0 0x7f069c145ef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f06980097a0 tx=0x7f069802b820 comp rx=0 tx=0).stop 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.790+0000 7f06a098a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.790+0000 7f06a3bf0640 1 -- 172.21.15.67:0/2341011122 shutdown_connections 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.790+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/2341011122 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c145ab0 0x7f069c145ef0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.790+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/2341011122 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c0a8a20 0x7f069c0a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.790+0000 7f06a3bf0640 1 -- 172.21.15.67:0/2341011122 >> 172.21.15.67:0/2341011122 conn(0x7f069c0b7890 msgr2=0x7f069c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.790+0000 7f06a3bf0640 1 -- 172.21.15.67:0/2341011122 shutdown_connections 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.790+0000 7f06a3bf0640 1 -- 172.21.15.67:0/2341011122 wait complete. 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 10 monclient: monmap: 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 1 librados: starting msgr at 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 1 librados: starting objecter 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 1 Processor -- start 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 1 -- start start 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 1 librados: setting wanted keys 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 1 librados: calling monclient init 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.791+0000 7f06a3bf0640 10 monclient: init 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c0a8a20 0x7f069c144750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): picked mon.noname-b con 0x7f069c0a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c13f0f0 0x7f069c131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): picked mon.noname-c con 0x7f069c13f0f0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f069c145ab0 0x7f069c131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): picked mon.noname-a con 0x7f069c145ab0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c1324c0 con 0x7f069c145ab0 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c1370a0 con 0x7f069c0a8a20 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c138340 con 0x7f069c13f0f0 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.792+0000 7f06a3bf0640 10 monclient(hunting): authenticate will time out at 1376.030648s 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.795+0000 7f04f95cf640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8075a10 0x7f04d8077e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 1 -- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc030006e80 msgr2=0x7fc030186c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc030006e80 0x7fc030186c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 1 -- 172.21.15.67:0/43435033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030030d40 msgr2=0x7fc0300d94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030030d40 0x7fc0300d94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 1 monclient: found mon.noname-a 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.797+0000 7fc03ca38640 1 -- 172.21.15.67:0/43435033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c008550 con 0x7fc030032180 2023-12-01T17:04:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.800+0000 7f06a218d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f069c145ab0 0x7f069c131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.802+0000 7f06a218d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f069c145ab0 0x7f069c131f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58008/0 (socket says 172.21.15.67:58008) 2023-12-01T17:04:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.802+0000 7f06a218d640 1 -- 172.21.15.67:0/1122042451 learned_addr learned my addr 172.21.15.67:0/1122042451 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.802+0000 7f06a218d640 10 monclient(hunting): get_auth_request con 0x7f069c145ab0 auth_method 0 2023-12-01T17:04:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.802+0000 7f06a218d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.802+0000 7f06a218d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.802+0000 7f06a218d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.805+0000 7f04f95cf640 10 monclient: get_auth_request con 0x7f04d8075a10 auth_method 0 2023-12-01T17:04:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.806+0000 7f06a218d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.806+0000 7f06a218d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.806+0000 7f06a218d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.808+0000 7fc03ca38640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc030032180 0x7fc030187330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc038000b60 tx=0x7fc038067960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.808+0000 7fc034ff9640 10 client.?.objecter ms_handle_connect 0x7fc030032180 2023-12-01T17:04:04.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.808+0000 7fc034ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 1 -- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c13f0f0 msgr2=0x7f069c131a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c13f0f0 0x7f069c131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 1 -- 172.21.15.67:0/1122042451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c0a8a20 msgr2=0x7f069c144750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c0a8a20 0x7f069c144750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 1 monclient: found mon.noname-a 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.814+0000 7f06a218d640 1 -- 172.21.15.67:0/1122042451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698009400 con 0x7f069c145ab0 2023-12-01T17:04:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.816+0000 7f04ea7fc640 10 client.4290.objecter ms_handle_connect 0x7f04d8075a10 2023-12-01T17:04:04.817 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.816+0000 7f04f95cf640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8075a10 0x7f04d8077e00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.819+0000 7f04ea7fc640 1 -- 172.21.15.67:0/1221287159 <== osd.1 v2:172.21.15.67:6826/866081726 1 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f04d8075720 con 0x7f04d8075a10 2023-12-01T17:04:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.819+0000 7f04ea7fc640 10 client.4290.objecter ms_dispatch 0x7f04f4000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:04:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.819+0000 7f04ea7fc640 10 client.4290.objecter _finish_command 2 = system:0 2023-12-01T17:04:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.819+0000 7f04ea7fc640 20 client.4290.objecter put_session s=0x7f04d8072580 osd=1 4 2023-12-01T17:04:04.821 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.819+0000 7f04ea7fc640 15 client.4290.objecter _session_command_op_remove 1 2 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 10 librados: watch_flush enter 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 10 librados: watch_flush exit 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 10 client.4290.objecter close_session for osd.1 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8075a10 msgr2=0x7f04d8077e00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f04d8075a10 0x7f04d8077e00 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 20 client.4290.objecter put_session s=0x7f04d8072580 osd=1 1 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 20 client.4290.objecter shutdown clearing up homeless session... 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 10 client.4290.objecter successfully canceled tick 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f04d806c870 msgr2=0x7f04d806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.822 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f04d806c870 0x7f04d806ed30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f04ec15aa10 tx=0x7f04f0008040 comp rx=0 tx=0).stop 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 10 monclient: shutdown 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 1 -- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0c7fd0 msgr2=0x7f04ec160660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04d7fff640 1 --2- 172.21.15.67:0/1221287159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f04ec0c7fd0 0x7f04ec160660 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f04f406a1d0 tx=0x7f04f4067960 comp rx=0 tx=0).stop 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.820+0000 7f04fa5d1640 1 -- 172.21.15.67:0/1221287159 reap_dead start 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.821+0000 7f04d7fff640 1 -- 172.21.15.67:0/1221287159 shutdown_connections 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.821+0000 7f04d7fff640 1 -- 172.21.15.67:0/1221287159 >> 172.21.15.67:0/1221287159 conn(0x7f04ec0062d0 msgr2=0x7f04ec037450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.823 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.821+0000 7f04d7fff640 1 -- 172.21.15.67:0/1221287159 shutdown_connections 2023-12-01T17:04:04.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.821+0000 7f04d7fff640 1 -- 172.21.15.67:0/1221287159 wait complete. 2023-12-01T17:04:04.824 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.821+0000 7f04d7fff640 1 librados: shutdown 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 1 -- 172.21.15.67:0/43435033 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc03806ce50 con 0x7fc030032180 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: dump: 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.826 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 1 -- 172.21.15.67:0/43435033 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc03804edb0 con 0x7fc030032180 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 1 -- 172.21.15.67:0/43435033 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038072390 con 0x7fc030032180 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc034ff9640 10 monclient: dump: 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.827 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 5 monclient: authenticate success, global_id 4302 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 10 monclient: _renew_subs 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0300d9a80 con 0x7fc030032180 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc0367fc640 4 set_mon_vals no callback set 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 10 monclient: _renew_subs 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0300db3e0 con 0x7fc030032180 2023-12-01T17:04:04.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.824+0000 7fc03d239640 1 librados: init done 2023-12-01T17:04:04.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.825+0000 7fc03d239640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.825+0000 7fc03d239640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.825+0000 7fc03d239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.826+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0300d8fc0 con 0x7fc030032180 2023-12-01T17:04:04.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.831+0000 7f06a218d640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f069c145ab0 0x7f069c131f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f069400caf0 tx=0x7f069400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.831+0000 7f06927fc640 10 client.?.objecter ms_handle_connect 0x7f069c145ab0 2023-12-01T17:04:04.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.831+0000 7f06927fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:04.840 INFO:teuthology.orchestra.run.smithi067.stdout:73014444039 2023-12-01T17:04:04.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-01T17:04:04.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.859+0000 7f84cd7fa640 1 -- 172.21.15.67:0/531762456 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f84bc058c90 con 0x7f84d00722c0 2023-12-01T17:04:04.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.859+0000 7f84cd7fa640 10 client.4286.objecter ms_dispatch 0x7f84d0000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.859+0000 7f84cd7fa640 3 client.4286.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.859+0000 7f84cd7fa640 3 client.4286.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.859+0000 7f84cd7fa640 20 client.4286.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.867+0000 7fc034ff9640 1 -- 172.21.15.67:0/43435033 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc0380724f0 con 0x7fc030032180 2023-12-01T17:04:04.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.868+0000 7fc034ff9640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc01c06c870 0x7fc01c06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.868+0000 7fc037fff640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc01c06c870 0x7fc01c06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.868+0000 7fc037fff640 10 monclient: get_auth_request con 0x7fc01c06c870 auth_method 0 2023-12-01T17:04:04.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.869+0000 7fc037fff640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc01c06c870 0x7fc01c06ed30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc02c03b000 tx=0x7fc02c032000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.869+0000 7fc034ff9640 10 client.4302.objecter ms_handle_connect 0x7fc01c06c870 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.876+0000 7f7fd279e640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.876+0000 7f7fd279e640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.876+0000 7f7fd279e640 10 monclient: monmap: 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.878603+0000 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.878603+0000 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.878 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.877+0000 7f3eafc21640 10 monclient: get_monmap_and_config 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.877+0000 7f3eafc21640 10 monclient: build_initial_monmap 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.877+0000 7f3eafc21640 10 monclient: monmap: 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:04.879 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:04.878845+0000 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:04.878845+0000 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:04.880 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.877+0000 7f7fd279e640 1 Processor -- start 2023-12-01T17:04:04.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.877+0000 7f7fd279e640 1 -- start start 2023-12-01T17:04:04.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.877+0000 7f7fd279e640 10 monclient: init 2023-12-01T17:04:04.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.878+0000 7f7fd279e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.878+0000 7f3eafc21640 1 Processor -- start 2023-12-01T17:04:04.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.878+0000 7f3eafc21640 1 -- start start 2023-12-01T17:04:04.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.878+0000 7f3eafc21640 10 monclient: init 2023-12-01T17:04:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.878+0000 7f7fd279e640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-12-01T17:04:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7fcc0a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7fcc1461c0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7fcc0a8e00 0x7f7fcc13f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fcc0a8e00 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-12-01T17:04:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3ea806a770 0x7f3ea8143460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient(hunting): picked mon.noname-a con 0x7f3ea806a770 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea81439a0 0x7f3ea8070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:04.883 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient(hunting): picked mon.noname-b con 0x7f3ea81439a0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ea8071470 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea806f380 con 0x7f3ea8071470 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ea806ab50 con 0x7f3ea806a770 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea806f640 con 0x7f3ea81439a0 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f3eafc21640 10 monclient(hunting): authenticate will time out at 1376.117850s 2023-12-01T17:04:04.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc145620 con 0x7f7fcc1461c0 2023-12-01T17:04:04.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fcc0aace0 con 0x7f7fcc0a8a20 2023-12-01T17:04:04.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:04.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fcc0aafc0 con 0x7f7fcc0a8e00 2023-12-01T17:04:04.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:04.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.879+0000 7f7fd279e640 10 monclient(hunting): authenticate will time out at 1376.118374s 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06927fc640 1 -- 172.21.15.67:0/1122042451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069400aca0 con 0x7f069c145ab0 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06927fc640 10 monclient: dump: 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.886 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06927fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06927fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 5 monclient: authenticate success, global_id 4305 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 10 monclient: _renew_subs 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f069c142f50 con 0x7f069c145ab0 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 10 monclient: _renew_subs 2023-12-01T17:04:04.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f069c1452b0 con 0x7f069c145ab0 2023-12-01T17:04:04.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.881+0000 7f06a3bf0640 1 librados: init done 2023-12-01T17:04:04.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.882+0000 7f06a3bf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.882+0000 7f06a3bf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.882+0000 7f06a3bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:04.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.882+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f069c138560 con 0x7f069c145ab0 2023-12-01T17:04:04.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.897+0000 7fc034ff9640 1 -- 172.21.15.67:0/43435033 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc0380c15b0 con 0x7fc030032180 2023-12-01T17:04:04.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.897+0000 7fc034ff9640 10 client.4302.objecter ms_dispatch 0x7fc038000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:04.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.897+0000 7fc034ff9640 3 client.4302.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:04.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.897+0000 7fc034ff9640 3 client.4302.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:04.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.898+0000 7fc034ff9640 20 client.4302.objecter dump_active .. 0 homeless 2023-12-01T17:04:04.959 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.958+0000 7fe7fca46640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-01T17:04:04.959 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.958+0000 7fe7fca46640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-01T17:04:04.959 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.958+0000 7fe7fca46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:04.959 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.958+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe7f804e840 con 0x7fe7f81434a0 2023-12-01T17:04:04.967 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.966+0000 7f3ead1bc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea81439a0 0x7f3ea8070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:04.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.979+0000 7f84cd7fa640 1 -- 172.21.15.67:0/531762456 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84bc0548d0 con 0x7f84d00722c0 2023-12-01T17:04:04.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.979+0000 7f84cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:04.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.979+0000 7f84cd7fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.980+0000 7f3ead1bc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea81439a0 0x7f3ea8070f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41970/0 (socket says 172.21.15.67:41970) 2023-12-01T17:04:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.980+0000 7f3ead1bc640 1 -- 172.21.15.67:0/2692403153 learned_addr learned my addr 172.21.15.67:0/2692403153 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.980+0000 7f3ead1bc640 10 monclient(hunting): get_auth_request con 0x7f3ea81439a0 auth_method 0 2023-12-01T17:04:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.980+0000 7f3ead1bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:04.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.980+0000 7f3ead1bc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:04.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.980+0000 7f3ead1bc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.982+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.982+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:04.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.982+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:04.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_done global_id 4289 payload 274 2023-12-01T17:04:04.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 1 -- 172.21.15.67:0/2692403153 >> v2:172.21.15.67:3300/0 conn(0x7f3ea806a770 msgr2=0x7f3ea8143460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 1 --2- 172.21.15.67:0/2692403153 >> v2:172.21.15.67:3300/0 conn(0x7f3ea806a770 0x7f3ea8143460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 1 -- 172.21.15.67:0/2692403153 >> v1:172.21.15.67:6789/0 conn(0x7f3ea8071470 legacy=0x7f3ea81498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 1 monclient: found mon.noname-b 2023-12-01T17:04:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.985+0000 7f3ead1bc640 1 -- 172.21.15.67:0/2692403153 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea8070630 con 0x7f3ea81439a0 2023-12-01T17:04:04.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.988+0000 7f3ead1bc640 1 --2- 172.21.15.67:0/2692403153 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea81439a0 0x7f3ea8070f30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009870 tx=0x7f3e9802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e581de3e4c47b579 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.992+0000 7f3eac9bb640 1 -- 172.21.15.67:0/2692403153 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98031640 con 0x7f3ea81439a0 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: dump: 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.994 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: _finish_auth 0 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: _check_auth_tickets 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 1 -- 172.21.15.67:0/2692403153 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e980317a0 con 0x7f3ea81439a0 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 5 monclient: authenticate success, global_id 4289 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:04.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 4 set_mon_vals no callback set 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 1 -- 172.21.15.67:0/2692403153 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e98031b30 con 0x7f3ea81439a0 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eac9bb640 10 monclient: dump: 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:04.996 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 10 monclient: shutdown 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 1 -- 172.21.15.67:0/2692403153 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea81439a0 msgr2=0x7f3ea8070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.993+0000 7f3eafc21640 1 --2- 172.21.15.67:0/2692403153 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea81439a0 0x7f3ea8070f30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009870 tx=0x7f3e9802b8f0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.994+0000 7f3eafc21640 1 -- 172.21.15.67:0/2692403153 shutdown_connections 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.994+0000 7f3eafc21640 1 --2- 172.21.15.67:0/2692403153 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea81439a0 0x7f3ea8070f30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.994+0000 7f3eafc21640 1 --2- 172.21.15.67:0/2692403153 >> v2:172.21.15.67:3300/0 conn(0x7f3ea806a770 0x7f3ea8143460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.994+0000 7f3eafc21640 1 -- 172.21.15.67:0/2692403153 >> 172.21.15.67:0/2692403153 conn(0x7f3ea80b1b60 msgr2=0x7f3ea80b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.994+0000 7f3eafc21640 1 -- 172.21.15.67:0/2692403153 shutdown_connections 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.994+0000 7f3eafc21640 1 -- 172.21.15.67:0/2692403153 wait complete. 2023-12-01T17:04:04.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 10 monclient: build_initial_monmap 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 10 monclient: monmap: 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 1 librados: starting msgr at 2023-12-01T17:04:05.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 1 librados: starting objecter 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 1 Processor -- start 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 1 -- start start 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 1 librados: setting wanted keys 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 1 librados: calling monclient init 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.995+0000 7f3eafc21640 10 monclient: init 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea806a770 0x7f3ea814a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 10 monclient(hunting): picked mon.noname-a con 0x7f3ea806a770 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea8071470 0x7f3ea80a89a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 10 monclient(hunting): picked mon.noname-c con 0x7f3ea8071470 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ea81439a0 0x7f3ea81dece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.996+0000 7f3eafc21640 10 monclient(hunting): picked mon.noname-b con 0x7f3ea81439a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:05.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80aa460 con 0x7f3ea806a770 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea81de7c0 con 0x7f3ea81439a0 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea8147990 con 0x7f3ea8071470 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3eafc21640 10 monclient(hunting): authenticate will time out at 1376.235428s 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3ead1bc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea8071470 0x7f3ea80a89a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.997+0000 7f3ead1bc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea8071470 0x7f3ea80a89a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:41984/0 (socket says 172.21.15.67:41984) 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 1 -- 172.21.15.67:0/1903580027 learned_addr learned my addr 172.21.15.67:0/1903580027 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 10 monclient(hunting): get_auth_request con 0x7f3ea8071470 auth_method 0 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:05.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:05.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:04.998+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 10 monclient(hunting): handle_auth_done global_id 4292 payload 824 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 1 -- 172.21.15.67:0/1903580027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ea81439a0 msgr2=0x7f3ea81dece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ea81439a0 0x7f3ea81dece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 1 -- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea806a770 msgr2=0x7f3ea814a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea806a770 0x7f3ea814a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 1 monclient: found mon.noname-c 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:05.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.001+0000 7f3ead1bc640 1 -- 172.21.15.67:0/1903580027 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e980094d0 con 0x7f3ea8071470 2023-12-01T17:04:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.002+0000 7fbc41ab4640 10 monclient: get_monmap_and_config 2023-12-01T17:04:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.002+0000 7fbc41ab4640 10 monclient: build_initial_monmap 2023-12-01T17:04:05.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.002+0000 7fbc41ab4640 10 monclient: monmap: 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:05.003810+0000 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:05.003810+0000 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.002+0000 7fbc41ab4640 1 Processor -- start 2023-12-01T17:04:05.006 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.002+0000 7fbc41ab4640 1 -- start start 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.002+0000 7fbc41ab4640 10 monclient: init 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc3c0a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbc3c1461c0 0x7fbc3c0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc3c1461c0 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbc3c13f0f0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3c145620 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3c0aace0 con 0x7fbc3c0a8a20 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3c13fc60 con 0x7fbc3c1461c0 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:05.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7fbc41ab4640 10 monclient(hunting): authenticate will time out at 1376.242118s 2023-12-01T17:04:05.009 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7f3ead1bc640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea8071470 0x7f3ea80a89a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009870 tx=0x7f3e980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.009 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.003+0000 7f3e9e7fc640 10 client.?.objecter ms_handle_connect 0x7f3ea8071470 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.004+0000 7f3e9e7fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3e9e7fc640 1 -- 172.21.15.67:0/1903580027 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9803ac60 con 0x7f3ea8071470 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3e9e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3e9e7fc640 10 monclient: dump: 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.010 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3e9e7fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3e9e7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3e9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 5 monclient: authenticate success, global_id 4292 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 10 monclient: _renew_subs 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea80a90f0 con 0x7f3ea8071470 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 10 monclient: _renew_subs 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea81df220 con 0x7f3ea8071470 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.006+0000 7f3eafc21640 1 librados: init done 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3eafc21640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3eafc21640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3eafc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea804e840 con 0x7f3ea8071470 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3e9e7fc640 1 -- 172.21.15.67:0/1903580027 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e98031e60 con 0x7f3ea8071470 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3e9e7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3e9e7fc640 1 -- 172.21.15.67:0/1903580027 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9804d650 con 0x7f3ea8071470 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3e9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3e9e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3e9e7fc640 10 monclient: dump: 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.012 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.013 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.013 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.013 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.008+0000 7f3e9ffff640 4 set_mon_vals no callback set 2023-12-01T17:04:05.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.029+0000 7fa699ffb640 1 -- 172.21.15.67:0/1865339612 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa69c01d3b0 con 0x7fa6a40ae100 2023-12-01T17:04:05.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.030+0000 7fa699ffb640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa68806c820 0x7fa68806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.030+0000 7fa6a9119640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa68806c820 0x7fa68806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.030+0000 7fa6a9119640 10 monclient: get_auth_request con 0x7fa68806c820 auth_method 0 2023-12-01T17:04:05.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.031+0000 7fa699ffb640 10 client.4336.objecter ms_handle_connect 0x7fa68806c820 2023-12-01T17:04:05.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.031+0000 7fa6a9119640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa68806c820 0x7fa68806ece0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa69402b8c0 tx=0x7fa69403d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.033+0000 7fa699ffb640 1 -- 172.21.15.67:0/1865339612 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa69c0589a0 con 0x7fa6a40ae100 2023-12-01T17:04:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.033+0000 7fa699ffb640 10 client.4336.objecter ms_dispatch 0x7fa6a4000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.033+0000 7fa699ffb640 3 client.4336.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.033+0000 7fa699ffb640 3 client.4336.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:05.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.033+0000 7fa699ffb640 20 client.4336.objecter dump_active .. 0 homeless 2023-12-01T17:04:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.042+0000 7fa699ffb640 1 -- 172.21.15.67:0/1865339612 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa69c054ce0 con 0x7fa6a40ae100 2023-12-01T17:04:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.042+0000 7fa699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.044 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.042+0000 7fa699ffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:05.048 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.047+0000 7f7fd0d3b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7fcc0a8e00 0x7f7fcc13f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.048+0000 7f7fd0d3b640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f7fcc0a8e00 0x7f7fcc13f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36660/0 (socket says 172.21.15.67:36660) 2023-12-01T17:04:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.048+0000 7f7fd0d3b640 1 -- 172.21.15.67:0/1879815408 learned_addr learned my addr 172.21.15.67:0/1879815408 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.048+0000 7f7fd0d3b640 10 monclient(hunting): get_auth_request con 0x7f7fcc0a8e00 auth_method 0 2023-12-01T17:04:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.048+0000 7f7fd0d3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.048+0000 7f7fd0d3b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:05.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.048+0000 7f7fd0d3b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.049+0000 7f7fd0d3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.049+0000 7f7fd0d3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:05.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.049+0000 7f7fd0d3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 10 monclient(hunting): handle_auth_done global_id 4345 payload 274 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 1 -- 172.21.15.67:0/1879815408 >> v1:172.21.15.92:6789/0 conn(0x7f7fcc0a8a20 legacy=0x7f7fcc145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 1 -- 172.21.15.67:0/1879815408 >> v1:172.21.15.67:6789/0 conn(0x7f7fcc1461c0 legacy=0x7f7fcc13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 10 monclient: _finish_hunting 0 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 1 monclient: found mon.noname-c 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.055+0000 7f7fd0d3b640 1 -- 172.21.15.67:0/1879815408 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fcc0b2210 con 0x7f7fcc0a8e00 2023-12-01T17:04:05.066 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.065+0000 7f7fd0d3b640 1 --2- 172.21.15.67:0/1879815408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8e00 0x7f7fcc13f800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009870 tx=0x7f7fbc02b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87de75cb1c974497 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.068+0000 7fe7e6ffd640 1 -- 172.21.15.67:0/4065097718 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe7ec021020 con 0x7fe7f81434a0 2023-12-01T17:04:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.068+0000 7fe7e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-01T17:04:05.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.068+0000 7fe7e6ffd640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:05.069 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2023-12-01T17:04:05.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.070+0000 7fe7fca46640 10 librados: watch_flush enter 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 10 librados: watch_flush exit 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 20 client.4330.objecter shutdown clearing up homeless session... 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 10 client.4330.objecter successfully canceled tick 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d806c820 msgr2=0x7fe7d806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d806c820 0x7fe7d806ece0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8000ff0 tx=0x7fe7e80014d0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 10 monclient: shutdown 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f81434a0 msgr2=0x7fe7f8143920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f81434a0 0x7fe7f8143920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00e840 tx=0x7fe7ec00ed00 comp rx=0 tx=0).stop 2023-12-01T17:04:05.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fbc3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbc3c1461c0 0x7fbc3c0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fbc3affd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbc3c1461c0 0x7fbc3c0a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36676/0 (socket says 172.21.15.67:36676) 2023-12-01T17:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fbc3affd640 1 -- 172.21.15.67:0/47761985 learned_addr learned my addr 172.21.15.67:0/47761985 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fbc3affd640 10 monclient(hunting): get_auth_request con 0x7fbc3c1461c0 auth_method 0 2023-12-01T17:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fbc3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fbc3affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fbc3affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:05.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 shutdown_connections 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f8135bd0 0x7fe7f8136050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f81434a0 0x7fe7f8143920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d806c820 0x7fe7d806ece0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 --2- 172.21.15.67:0/4065097718 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7e0004e20 0x7fe7f813a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 >> 172.21.15.67:0/4065097718 conn(0x7fe7f8061a10 msgr2=0x7fe7f813f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 shutdown_connections 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 -- 172.21.15.67:0/4065097718 wait complete. 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.071+0000 7fe7fca46640 1 librados: shutdown 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.072+0000 7fbc3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.072+0000 7fbc3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:05.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.072+0000 7fbc3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 1 -- 172.21.15.67:0/47761985 >> v1:172.21.15.92:6789/0 conn(0x7fbc3c0a8a20 legacy=0x7fbc3c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 1 -- 172.21.15.67:0/47761985 >> v1:172.21.15.67:6789/0 conn(0x7fbc3c13f0f0 legacy=0x7fbc3c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 1 monclient: found mon.noname-c 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.077 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.074+0000 7fbc3affd640 1 -- 172.21.15.67:0/47761985 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3c0aafc0 con 0x7fbc3c1461c0 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 1 -- 172.21.15.67:0/1879815408 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc030860 con 0x7f7fcc0a8e00 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: dump: 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.079 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: _finish_auth 0 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 1 -- 172.21.15.67:0/1879815408 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fbc0309c0 con 0x7f7fcc0a8e00 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 1 -- 172.21.15.67:0/1879815408 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fbc030d50 con 0x7f7fcc0a8e00 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fcaffd640 10 monclient: dump: 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.080 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fd279e640 5 monclient: authenticate success, global_id 4345 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fd279e640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.076+0000 7f7fd279e640 4 set_mon_vals no callback set 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 10 monclient: shutdown 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 1 -- 172.21.15.67:0/1879815408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8e00 msgr2=0x7f7fcc13f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 1 --2- 172.21.15.67:0/1879815408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8e00 0x7f7fcc13f800 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc009870 tx=0x7f7fbc02b9a0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 1 -- 172.21.15.67:0/1879815408 shutdown_connections 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 1 --2- 172.21.15.67:0/1879815408 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8e00 0x7f7fcc13f800 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 1 -- 172.21.15.67:0/1879815408 >> 172.21.15.67:0/1879815408 conn(0x7f7fcc0b7890 msgr2=0x7f7fcc0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 1 -- 172.21.15.67:0/1879815408 shutdown_connections 2023-12-01T17:04:05.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.077+0000 7f7fd279e640 1 -- 172.21.15.67:0/1879815408 wait complete. 2023-12-01T17:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.078+0000 7f7fd279e640 10 monclient: build_initial_monmap 2023-12-01T17:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.078+0000 7f7fd279e640 10 monclient: monmap: 2023-12-01T17:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:05.082 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7f7fd279e640 1 librados: starting msgr at 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7f7fd279e640 1 librados: starting objecter 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7f7fd279e640 1 Processor -- start 2023-12-01T17:04:05.083 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7fbc3affd640 1 --2- 172.21.15.67:0/47761985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c1461c0 0x7fbc3c0a8e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc3002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6312f702d7e35c17 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7f7fd279e640 1 -- start start 2023-12-01T17:04:05.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7f7fd279e640 1 librados: setting wanted keys 2023-12-01T17:04:05.084 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7f7fd279e640 1 librados: calling monclient init 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.079+0000 7f7fd279e640 10 monclient: init 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.080+0000 7f7fd279e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.080+0000 7f7fd279e640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.080+0000 7f7fd279e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8a20 0x7f7fcc0b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.080+0000 7f7fd279e640 10 monclient(hunting): picked mon.noname-b con 0x7f7fcc0a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.080+0000 7f7fd279e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc0a8e00 0x7f7fcc13fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 10 monclient(hunting): picked mon.noname-a con 0x7f7fcc0a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fcc1461c0 0x7f7fcc1428e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 10 monclient(hunting): picked mon.noname-c con 0x7f7fcc1461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc141a90 con 0x7f7fcc0a8e00 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc141d70 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fcc142050 con 0x7f7fcc1461c0 2023-12-01T17:04:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.081+0000 7f7fd279e640 10 monclient(hunting): authenticate will time out at 1376.319605s 2023-12-01T17:04:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 1 -- 172.21.15.67:0/47761985 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031870 con 0x7fbc3c1461c0 2023-12-01T17:04:05.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: dump: 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:05.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7f7fcbfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8a20 0x7f7fcc0b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 1 -- 172.21.15.67:0/47761985 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc300319d0 con 0x7fbc3c1461c0 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 1 -- 172.21.15.67:0/47761985 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031d60 con 0x7fbc3c1461c0 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8a20 0x7f7fcc0b6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36690/0 (socket says 172.21.15.67:36690) 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 1 -- 172.21.15.67:0/967555924 learned_addr learned my addr 172.21.15.67:0/967555924 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 10 monclient(hunting): get_auth_request con 0x7f7fcc0a8a20 auth_method 0 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:05.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc3a7fc640 10 monclient: dump: 2023-12-01T17:04:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.090 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc41ab4640 5 monclient: authenticate success, global_id 4348 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc41ab4640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.083+0000 7fbc41ab4640 4 set_mon_vals no callback set 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 10 monclient: shutdown 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:05.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:05.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7f7fcbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 1 -- 172.21.15.67:0/47761985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c1461c0 msgr2=0x7fbc3c0a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/47761985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c1461c0 0x7fbc3c0a8e00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc3002b910 comp rx=0 tx=0).stop 2023-12-01T17:04:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 1 -- 172.21.15.67:0/47761985 shutdown_connections 2023-12-01T17:04:05.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/47761985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c1461c0 0x7fbc3c0a8e00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 1 -- 172.21.15.67:0/47761985 >> 172.21.15.67:0/47761985 conn(0x7fbc3c0b7890 msgr2=0x7fbc3c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 1 -- 172.21.15.67:0/47761985 shutdown_connections 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.084+0000 7fbc41ab4640 1 -- 172.21.15.67:0/47761985 wait complete. 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.085+0000 7fbc41ab4640 10 monclient: build_initial_monmap 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.086+0000 7fbc41ab4640 10 monclient: monmap: 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:05.094 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.086+0000 7fbc41ab4640 1 librados: starting msgr at 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.086+0000 7fbc41ab4640 1 librados: starting objecter 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.086+0000 7fbc41ab4640 1 Processor -- start 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7fbc41ab4640 1 -- start start 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7fbc41ab4640 1 librados: setting wanted keys 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7fbc41ab4640 1 librados: calling monclient init 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7fbc41ab4640 10 monclient: init 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 1 -- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fcc1461c0 msgr2=0x7f7fcc1428e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fcc1461c0 0x7f7fcc1428e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 1 -- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc0a8e00 msgr2=0x7f7fcc13fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc0a8e00 0x7f7fcc13fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 1 monclient: found mon.noname-b 2023-12-01T17:04:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.087+0000 7f7fcbfff640 1 -- 172.21.15.67:0/967555924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc0094d0 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.5 2023-12-01T17:04:05.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc3c0a8a20 0x7fbc3c144f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 10 monclient(hunting): picked mon.noname-a con 0x7fbc3c0a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c13f0f0 0x7fbc3c140200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 10 monclient(hunting): picked mon.noname-b con 0x7fbc3c13f0f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbc3c1461c0 0x7fbc3c0b6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 10 monclient(hunting): picked mon.noname-c con 0x7fbc3c1461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.088+0000 7fbc41ab4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.089+0000 7fbc41ab4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c13fdb0 con 0x7fbc3c0a8a20 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.089+0000 7fbc41ab4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.089+0000 7fbc41ab4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c141b00 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.089+0000 7fbc41ab4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.089+0000 7fbc41ab4640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3c141d70 con 0x7fbc3c1461c0 2023-12-01T17:04:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.089+0000 7fbc41ab4640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.089+0000 7fbc41ab4640 10 monclient(hunting): authenticate will time out at 1376.327538s 2023-12-01T17:04:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.090+0000 7f7fcbfff640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8a20 0x7f7fcc0b6e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c870 tx=0x7f7fc400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.090+0000 7f7fc8ff9640 10 client.?.objecter ms_handle_connect 0x7f7fcc0a8a20 2023-12-01T17:04:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.090+0000 7f7fc8ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:05.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7fbc3affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c13f0f0 0x7fbc3c140200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 1 -- 172.21.15.67:0/967555924 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4003b70 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: dump: 2023-12-01T17:04:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.101 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: _finish_auth 0 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 1 -- 172.21.15.67:0/967555924 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc4003cf0 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 5 monclient: authenticate success, global_id 4351 2023-12-01T17:04:05.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 1 -- 172.21.15.67:0/967555924 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc400aa90 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fc8ff9640 10 monclient: dump: 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 10 monclient: _renew_subs 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fca7fc640 4 set_mon_vals no callback set 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fcc140980 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 10 monclient: _renew_subs 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fcc140cf0 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7f7fd279e640 1 librados: init done 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7fbc3affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c13f0f0 0x7fbc3c140200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36696/0 (socket says 172.21.15.67:36696) 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7fbc3affd640 1 -- 172.21.15.67:0/615410259 learned_addr learned my addr 172.21.15.67:0/615410259 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7fbc3affd640 10 monclient(hunting): get_auth_request con 0x7fbc3c13f0f0 auth_method 0 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7fbc3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7fbc3affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.094+0000 7fbc3affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.096+0000 7f7fd279e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.096+0000 7f7fd279e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.096+0000 7f7fd279e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.096+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fcc04e840 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.103+0000 7fbc3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.103+0000 7fbc3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:05.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.103+0000 7fbc3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 1 -- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbc3c1461c0 msgr2=0x7fbc3c0b6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbc3c1461c0 0x7fbc3c0b6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 1 -- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc3c0a8a20 msgr2=0x7fbc3c144f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc3c0a8a20 0x7fbc3c144f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 1 monclient: found mon.noname-b 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.107+0000 7fbc3affd640 1 -- 172.21.15.67:0/615410259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc300094d0 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.121+0000 7f7fc8ff9640 1 -- 172.21.15.67:0/967555924 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7fc400abf0 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.121+0000 7f7fc8ff9640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f7fb806c850 0x7f7fb806ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.122+0000 7f7fcb7fe640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f7fb806c850 0x7f7fb806ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.122+0000 7f7fcb7fe640 10 monclient: get_auth_request con 0x7f7fb806c850 auth_method 0 2023-12-01T17:04:05.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.122+0000 7f7fc8ff9640 10 client.4351.objecter ms_handle_connect 0x7f7fb806c850 2023-12-01T17:04:05.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.122+0000 7f7fcb7fe640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f7fb806c850 0x7f7fb806ed10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0b6b90 tx=0x7f7fc0006ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.124+0000 7f7fc8ff9640 1 -- 172.21.15.67:0/967555924 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7fc4059420 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.124+0000 7f7fc8ff9640 10 client.4351.objecter ms_dispatch 0x7f7fcc000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.124+0000 7f7fc8ff9640 3 client.4351.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.124+0000 7f7fc8ff9640 3 client.4351.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.124+0000 7f7fc8ff9640 20 client.4351.objecter dump_active .. 0 homeless 2023-12-01T17:04:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.125+0000 7fbc3affd640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c13f0f0 0x7fbc3c140200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.125+0000 7fbc23fff640 10 client.?.objecter ms_handle_connect 0x7fbc3c13f0f0 2023-12-01T17:04:05.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.125+0000 7fbc23fff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:05.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.177+0000 7f84d6df0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-01T17:04:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.177+0000 7f84d6df0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-01T17:04:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.177+0000 7f84d6df0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:05.179 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.177+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f84d0138ec0 con 0x7f84d00722c0 2023-12-01T17:04:05.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.227+0000 7fa6ab37d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-01T17:04:05.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.227+0000 7fa6ab37d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-01T17:04:05.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.227+0000 7fa6ab37d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.227+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa6a412f140 con 0x7fa6a40ae100 2023-12-01T17:04:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.287+0000 7f3e9e7fc640 1 -- 172.21.15.67:0/1903580027 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3e9802f050 con 0x7f3ea8071470 2023-12-01T17:04:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.288+0000 7f3e9e7fc640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e8c06c820 0x7f3e8c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.288+0000 7f3ead9bd640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e8c06c820 0x7f3e8c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.288+0000 7f3ead9bd640 10 monclient: get_auth_request con 0x7f3e8c06c820 auth_method 0 2023-12-01T17:04:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.288+0000 7f3e9e7fc640 10 client.4292.objecter ms_handle_connect 0x7f3e8c06c820 2023-12-01T17:04:05.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.288+0000 7f3ead9bd640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e8c06c820 0x7f3e8c06ece0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009710 tx=0x7f3ea4006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.300+0000 7f3e9e7fc640 1 -- 172.21.15.67:0/1903580027 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3e9807b050 con 0x7f3ea8071470 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.300+0000 7f3e9e7fc640 10 client.4292.objecter ms_dispatch 0x7f3ea8000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.300+0000 7f3e9e7fc640 3 client.4292.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.300+0000 7f3e9e7fc640 3 client.4292.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.301+0000 7f3e9e7fc640 20 client.4292.objecter dump_active .. 0 homeless 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.301+0000 7f3e9e7fc640 1 -- 172.21.15.67:0/1903580027 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e98076c90 con 0x7f3ea8071470 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.301+0000 7f3e9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.301+0000 7f3e9e7fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:05.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.310+0000 7f84cd7fa640 1 -- 172.21.15.67:0/531762456 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f84d0138ec0 con 0x7f84d00722c0 2023-12-01T17:04:05.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.311+0000 7f84cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-01T17:04:05.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.311+0000 7f84cd7fa640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:05.312 INFO:teuthology.orchestra.run.smithi067.stdout:60129542147 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 10 librados: watch_flush enter 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 10 librados: watch_flush exit 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 20 client.4286.objecter shutdown clearing up homeless session... 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 10 client.4286.objecter successfully canceled tick 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f84b406c870 msgr2=0x7f84b406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f84b406c870 0x7f84b406ed30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f84c4001560 tx=0x7f84c4007990 comp rx=0 tx=0).stop 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 10 monclient: shutdown 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d00722c0 msgr2=0x7f84d00acb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d00722c0 0x7f84d00acb40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f84bc00b4d0 tx=0x7f84bc00b990 comp rx=0 tx=0).stop 2023-12-01T17:04:05.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 shutdown_connections 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f84d0141ea0 0x7f84d0137d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f84b406c870 0x7f84b406ed30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f84d00722c0 0x7f84d00acb40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 --2- 172.21.15.67:0/531762456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f84d0071ee0 0x7f84d01391a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 >> 172.21.15.67:0/531762456 conn(0x7f84d00b3e50 msgr2=0x7f84d00b1640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 shutdown_connections 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 -- 172.21.15.67:0/531762456 wait complete. 2023-12-01T17:04:05.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.313+0000 7f84d6df0640 1 librados: shutdown 2023-12-01T17:04:05.326 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542147 for osd.6 2023-12-01T17:04:05.356 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.355+0000 7f7fc8ff9640 1 -- 172.21.15.67:0/967555924 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fc4055060 con 0x7f7fcc0a8a20 2023-12-01T17:04:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.355+0000 7f7fc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.357 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.355+0000 7f7fc8ff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 1 -- 172.21.15.67:0/615410259 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3003ad30 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: dump: 2023-12-01T17:04:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.366 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: _finish_auth 0 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 1 -- 172.21.15.67:0/615410259 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3003ae90 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 5 monclient: authenticate success, global_id 4354 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 10 monclient: _renew_subs 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc3c0b70a0 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 1 -- 172.21.15.67:0/615410259 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30044630 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc23fff640 10 monclient: dump: 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.368 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 10 monclient: _renew_subs 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc3c142370 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc41ab4640 1 librados: init done 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.364+0000 7fbc39ffb640 4 set_mon_vals no callback set 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.365+0000 7fa699ffb640 1 -- 172.21.15.67:0/1865339612 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa69c019020 con 0x7fa6a40ae100 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.365+0000 7fa699ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-01T17:04:05.369 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.365+0000 7fa699ffb640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:05.370 INFO:teuthology.orchestra.run.smithi067.stdout:68719476741 2023-12-01T17:04:05.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.366+0000 7fbc41ab4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.366+0000 7fbc41ab4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.366+0000 7fbc41ab4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:05.371 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.366+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3c0b7230 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 10 librados: watch_flush enter 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 10 librados: watch_flush exit 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 20 client.4336.objecter shutdown clearing up homeless session... 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 10 client.4336.objecter successfully canceled tick 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa68806c820 msgr2=0x7fa68806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa68806c820 0x7fa68806ece0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa69402b8c0 tx=0x7fa69403d040 comp rx=0 tx=0).stop 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 10 monclient: shutdown 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:05.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a40ae100 msgr2=0x7fa6a412e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.367+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a40ae100 0x7fa6a412e7a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa69c00bd70 tx=0x7fa69c00be60 comp rx=0 tx=0).stop 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 shutdown_connections 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa68806c820 0x7fa68806ece0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa6a413ec40 0x7fa6a41299b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa6a40ae100 0x7fa6a412e7a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 --2- 172.21.15.67:0/1865339612 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a406f800 0x7fa6a4141090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 >> 172.21.15.67:0/1865339612 conn(0x7fa6a40b3b20 msgr2=0x7fa6a406eef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 shutdown_connections 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 -- 172.21.15.67:0/1865339612 wait complete. 2023-12-01T17:04:05.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.368+0000 7fa6ab37d640 1 librados: shutdown 2023-12-01T17:04:05.383 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476741 for osd.3 2023-12-01T17:04:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.384+0000 7fbc23fff640 1 -- 172.21.15.67:0/615410259 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbc3004d450 con 0x7fbc3c13f0f0 2023-12-01T17:04:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.384+0000 7fbc23fff640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbc2806c870 0x7fbc2806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.384+0000 7fbc3b7fe640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbc2806c870 0x7fbc2806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.385 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.384+0000 7fbc3b7fe640 10 monclient: get_auth_request con 0x7fbc2806c870 auth_method 0 2023-12-01T17:04:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.385+0000 7fbc23fff640 10 client.4354.objecter ms_handle_connect 0x7fbc2806c870 2023-12-01T17:04:05.386 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.385+0000 7fbc3b7fe640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbc2806c870 0x7fbc2806ed30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009650 tx=0x7fbc2c006ba0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.428+0000 7fc034ff9640 1 -- 172.21.15.67:0/43435033 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc0380bd1f0 con 0x7fc030032180 2023-12-01T17:04:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.428+0000 7fc034ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.428+0000 7fc034ff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.458+0000 7f06927fc640 1 -- 172.21.15.67:0/1122042451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0694005550 con 0x7f069c145ab0 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.458+0000 7f06927fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.458+0000 7f06927fc640 1 -- 172.21.15.67:0/1122042451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694025430 con 0x7f069c145ab0 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.458+0000 7f06927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.458+0000 7f06927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.458+0000 7f06927fc640 10 monclient: dump: 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.460 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:05.461 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:05.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.458+0000 7f0693fff640 4 set_mon_vals no callback set 2023-12-01T17:04:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.459+0000 7f06927fc640 1 -- 172.21.15.67:0/1122042451 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f0694025650 con 0x7f069c145ab0 2023-12-01T17:04:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.460+0000 7f06927fc640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f068006c820 0x7f068006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.460+0000 7f06a198c640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f068006c820 0x7f068006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.460+0000 7f06a198c640 10 monclient: get_auth_request con 0x7f068006c820 auth_method 0 2023-12-01T17:04:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.461+0000 7f06a198c640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f068006c820 0x7f068006ece0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f068c009710 tx=0x7f068c006c90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:05.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.461+0000 7f06927fc640 10 client.4305.objecter ms_handle_connect 0x7f068006c820 2023-12-01T17:04:05.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 1 -- 172.21.15.67:0/1122042451 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f0694059b50 con 0x7f069c145ab0 2023-12-01T17:04:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 10 client.4305.objecter ms_dispatch 0x7f069c000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 3 client.4305.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 3 client.4305.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 20 client.4305.objecter dump_active .. 0 homeless 2023-12-01T17:04:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 1 -- 172.21.15.67:0/1122042451 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0694055790 con 0x7f069c145ab0 2023-12-01T17:04:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:05.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.474+0000 7f06927fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:05.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.507+0000 7f3eafc21640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-01T17:04:06.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.507+0000 7f3eafc21640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-01T17:04:06.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.507+0000 7f3eafc21640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:06.025 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.507+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f3ea8147ad0 con 0x7f3ea8071470 2023-12-01T17:04:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.513+0000 7f3e9e7fc640 1 -- 172.21.15.67:0/1903580027 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3e9807a2b0 con 0x7f3ea8071470 2023-12-01T17:04:06.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.514+0000 7f3e9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-01T17:04:06.026 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2023-12-01T17:04:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.514+0000 7f3e9e7fc640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 10 librados: watch_flush enter 2023-12-01T17:04:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 10 librados: watch_flush exit 2023-12-01T17:04:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 20 client.4292.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 10 client.4292.objecter successfully canceled tick 2023-12-01T17:04:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e8c06c820 msgr2=0x7f3e8c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e8c06c820 0x7f3e8c06ece0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009710 tx=0x7f3ea4006c90 comp rx=0 tx=0).stop 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 10 monclient: shutdown 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea8071470 msgr2=0x7f3ea80a89a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea8071470 0x7f3ea80a89a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009870 tx=0x7f3e980046c0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 shutdown_connections 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3ea81439a0 0x7f3ea81dece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.516+0000 7f3eafc21640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3e8c06c820 0x7f3e8c06ece0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.517+0000 7f3eafc21640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3ea8071470 0x7f3ea80a89a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.517+0000 7f3eafc21640 1 --2- 172.21.15.67:0/1903580027 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ea806a770 0x7f3ea814a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.517+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 >> 172.21.15.67:0/1903580027 conn(0x7f3ea80b1b60 msgr2=0x7f3ea8146450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.517+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 shutdown_connections 2023-12-01T17:04:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.517+0000 7f3eafc21640 1 -- 172.21.15.67:0/1903580027 wait complete. 2023-12-01T17:04:06.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.517+0000 7f3eafc21640 1 librados: shutdown 2023-12-01T17:04:06.031 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.4 2023-12-01T17:04:06.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.562+0000 7f7fd279e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-01T17:04:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.562+0000 7f7fd279e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-01T17:04:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.562+0000 7f7fd279e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.562+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7fcc141eb0 con 0x7f7fcc0a8a20 2023-12-01T17:04:06.033 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.611+0000 7fbc23fff640 1 -- 172.21.15.67:0/615410259 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbc3002d040 con 0x7fbc3c13f0f0 2023-12-01T17:04:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.611+0000 7fbc23fff640 10 client.4354.objecter ms_dispatch 0x7fbc3c000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.611+0000 7fbc23fff640 3 client.4354.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.611+0000 7fbc23fff640 3 client.4354.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.611+0000 7fbc23fff640 20 client.4354.objecter dump_active .. 0 homeless 2023-12-01T17:04:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.622+0000 7fbc23fff640 1 -- 172.21.15.67:0/615410259 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc300772b0 con 0x7fbc3c13f0f0 2023-12-01T17:04:06.034 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.622+0000 7fbc23fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.622+0000 7fbc23fff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.624+0000 7fc03d239640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-01T17:04:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.624+0000 7fc03d239640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-01T17:04:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.624+0000 7fc03d239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.624+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc0300052c0 con 0x7fc030032180 2023-12-01T17:04:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.632+0000 7fc034ff9640 1 -- 172.21.15.67:0/43435033 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc0380c0810 con 0x7fc030032180 2023-12-01T17:04:06.036 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.632+0000 7fc034ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-01T17:04:06.037 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.632+0000 7fc034ff9640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.037 INFO:teuthology.orchestra.run.smithi067.stdout:55834574852 2023-12-01T17:04:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 10 librados: watch_flush enter 2023-12-01T17:04:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 10 librados: watch_flush exit 2023-12-01T17:04:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 20 client.4302.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.040 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 10 client.4302.objecter successfully canceled tick 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc01c06c870 msgr2=0x7fc01c06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc01c06c870 0x7fc01c06ed30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc02c03b000 tx=0x7fc02c032000 comp rx=0 tx=0).stop 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 10 monclient: shutdown 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc030032180 msgr2=0x7fc030187330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc030032180 0x7fc030187330 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc038000b60 tx=0x7fc038067960 comp rx=0 tx=0).stop 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 shutdown_connections 2023-12-01T17:04:06.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc030032180 0x7fc030187330 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc030030d40 0x7fc0300d94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc01c06c870 0x7fc01c06ed30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.635+0000 7fc03d239640 1 --2- 172.21.15.67:0/43435033 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc030006e80 0x7fc030186c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.636+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 >> 172.21.15.67:0/43435033 conn(0x7fc030005ee0 msgr2=0x7fc0300e2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.636+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 shutdown_connections 2023-12-01T17:04:06.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.636+0000 7fc03d239640 1 -- 172.21.15.67:0/43435033 wait complete. 2023-12-01T17:04:06.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.636+0000 7fc03d239640 1 librados: shutdown 2023-12-01T17:04:06.043 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.7 2023-12-01T17:04:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.688+0000 7f7fc8ff9640 1 -- 172.21.15.67:0/967555924 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7fcc141eb0 con 0x7f7fcc0a8a20 2023-12-01T17:04:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.688+0000 7f7fc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-01T17:04:06.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.688+0000 7f7fc8ff9640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.044 INFO:teuthology.orchestra.run.smithi067.stdout:64424509443 2023-12-01T17:04:06.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.691+0000 7f7fd279e640 10 librados: watch_flush enter 2023-12-01T17:04:06.047 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.691+0000 7f7fd279e640 10 librados: watch_flush exit 2023-12-01T17:04:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.691+0000 7f7fd279e640 20 client.4351.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.691+0000 7f7fd279e640 10 client.4351.objecter successfully canceled tick 2023-12-01T17:04:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f7fb806c850 msgr2=0x7f7fb806ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f7fb806c850 0x7f7fb806ed10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc0b6b90 tx=0x7f7fc0006ba0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.048 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 10 monclient: shutdown 2023-12-01T17:04:06.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8a20 msgr2=0x7f7fcc0b6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8a20 0x7f7fcc0b6e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c870 tx=0x7f7fc400cd30 comp rx=0 tx=0).stop 2023-12-01T17:04:06.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 shutdown_connections 2023-12-01T17:04:06.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f7fb806c850 0x7f7fb806ed10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f7fcc1461c0 0x7f7fcc1428e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7fcc0a8e00 0x7f7fcc13fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 --2- 172.21.15.67:0/967555924 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f7fcc0a8a20 0x7f7fcc0b6e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 >> 172.21.15.67:0/967555924 conn(0x7f7fcc0b7890 msgr2=0x7f7fcc139770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 shutdown_connections 2023-12-01T17:04:06.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 -- 172.21.15.67:0/967555924 wait complete. 2023-12-01T17:04:06.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.692+0000 7f7fd279e640 1 librados: shutdown 2023-12-01T17:04:06.051 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509443 for osd.0 2023-12-01T17:04:06.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.719+0000 7f06a3bf0640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-01T17:04:06.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.719+0000 7f06a3bf0640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-01T17:04:06.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.719+0000 7f06a3bf0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:06.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.719+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f069c04e840 con 0x7f069c145ab0 2023-12-01T17:04:06.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.783+0000 7f06927fc640 1 -- 172.21.15.67:0/1122042451 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0694058db0 con 0x7f069c145ab0 2023-12-01T17:04:06.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.783+0000 7f06927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-01T17:04:06.053 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.783+0000 7f06927fc640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.053 INFO:teuthology.orchestra.run.smithi067.stdout:68719476741 2023-12-01T17:04:06.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 10 librados: watch_flush enter 2023-12-01T17:04:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 10 librados: watch_flush exit 2023-12-01T17:04:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 20 client.4305.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 10 client.4305.objecter successfully canceled tick 2023-12-01T17:04:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f068006c820 msgr2=0x7f068006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f068006c820 0x7f068006ece0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f068c009710 tx=0x7f068c006c90 comp rx=0 tx=0).stop 2023-12-01T17:04:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 10 monclient: shutdown 2023-12-01T17:04:06.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f069c145ab0 msgr2=0x7f069c131f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f069c145ab0 0x7f069c131f80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f069400caf0 tx=0x7f069400a060 comp rx=0 tx=0).stop 2023-12-01T17:04:06.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 shutdown_connections 2023-12-01T17:04:06.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f069c145ab0 0x7f069c131f80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f068006c820 0x7f068006ece0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f069c13f0f0 0x7f069c131a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 --2- 172.21.15.67:0/1122042451 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f069c0a8a20 0x7f069c144750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 >> 172.21.15.67:0/1122042451 conn(0x7f069c0b7890 msgr2=0x7f069c13cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 shutdown_connections 2023-12-01T17:04:06.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 -- 172.21.15.67:0/1122042451 wait complete. 2023-12-01T17:04:06.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.786+0000 7f06a3bf0640 1 librados: shutdown 2023-12-01T17:04:06.059 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476741 for osd.2 2023-12-01T17:04:06.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.823+0000 7fbc41ab4640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-01T17:04:06.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.823+0000 7fbc41ab4640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-01T17:04:06.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.823+0000 7fbc41ab4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.823+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbc3c0b7230 con 0x7fbc3c13f0f0 2023-12-01T17:04:06.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.827+0000 7fbc23fff640 1 -- 172.21.15.67:0/615410259 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbc3007a8d0 con 0x7fbc3c13f0f0 2023-12-01T17:04:06.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.827+0000 7fbc23fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-01T17:04:06.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.827+0000 7fbc23fff640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.060 INFO:teuthology.orchestra.run.smithi067.stdout:73014444037 2023-12-01T17:04:06.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 10 librados: watch_flush enter 2023-12-01T17:04:06.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 10 librados: watch_flush exit 2023-12-01T17:04:06.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 20 client.4354.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 10 client.4354.objecter successfully canceled tick 2023-12-01T17:04:06.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbc2806c870 msgr2=0x7fbc2806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbc2806c870 0x7fbc2806ed30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009650 tx=0x7fbc2c006ba0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 10 monclient: shutdown 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c13f0f0 msgr2=0x7fbc3c140200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c13f0f0 0x7fbc3c140200 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fbc30009870 tx=0x7fbc300046c0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 shutdown_connections 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbc2806c870 0x7fbc2806ed30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbc3c1461c0 0x7fbc3c0b6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbc3c13f0f0 0x7fbc3c140200 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 --2- 172.21.15.67:0/615410259 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbc3c0a8a20 0x7fbc3c144f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 >> 172.21.15.67:0/615410259 conn(0x7fbc3c0b7890 msgr2=0x7fbc3c0b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 shutdown_connections 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.829+0000 7fbc41ab4640 1 -- 172.21.15.67:0/615410259 wait complete. 2023-12-01T17:04:06.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:05.830+0000 7fbc41ab4640 1 librados: shutdown 2023-12-01T17:04:06.063 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444039 got 73014444037 for osd.1 2023-12-01T17:04:06.098 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-12-01T17:04:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.205+0000 7f3c0e77c640 10 monclient: get_monmap_and_config 2023-12-01T17:04:06.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.205+0000 7f3c0e77c640 10 monclient: build_initial_monmap 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.205+0000 7f3c0e77c640 10 monclient: monmap: 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:06.207393+0000 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:06.207393+0000 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:06.207 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.206+0000 7f3c0e77c640 1 Processor -- start 2023-12-01T17:04:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.206+0000 7f3c0e77c640 1 -- start start 2023-12-01T17:04:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.206+0000 7f3c0e77c640 10 monclient: init 2023-12-01T17:04:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.206+0000 7f3c0e77c640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.206+0000 7f3c0e77c640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-12-01T17:04:06.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3c080a8a20 0x7f3c080a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c080a8a20 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3c08145ab0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c0813f800 0x7f3c08145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c0813f800 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c08146430 con 0x7f3c08145ab0 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c08145620 con 0x7f3c0813f800 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c080aace0 con 0x7f3c080a8a20 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0e77c640 10 monclient(hunting): authenticate will time out at 1377.445465s 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c0cd19640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c0813f800 0x7f3c08145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.207+0000 7f3c07fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3c080a8a20 0x7f3c080a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3c080a8a20 0x7f3c080a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36708/0 (socket says 172.21.15.67:36708) 2023-12-01T17:04:06.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3c0813f800 0x7f3c08145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58076/0 (socket says 172.21.15.67:58076) 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 1 -- 172.21.15.67:0/849005692 learned_addr learned my addr 172.21.15.67:0/849005692 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 10 monclient(hunting): get_auth_request con 0x7f3c080a8a20 auth_method 0 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 10 monclient(hunting): get_auth_request con 0x7f3c0813f800 auth_method 0 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.208+0000 7f3c0cd19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 10 monclient(hunting): handle_auth_done global_id 4357 payload 274 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 1 -- 172.21.15.67:0/849005692 >> v2:172.21.15.67:3300/0 conn(0x7f3c0813f800 msgr2=0x7f3c08145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 1 --2- 172.21.15.67:0/849005692 >> v2:172.21.15.67:3300/0 conn(0x7f3c0813f800 0x7f3c08145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 1 -- 172.21.15.67:0/849005692 >> v1:172.21.15.67:6789/0 conn(0x7f3c08145ab0 legacy=0x7f3c0813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:06.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 1 monclient: found mon.noname-c 2023-12-01T17:04:06.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:06.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.210+0000 7f3c07fff640 1 -- 172.21.15.67:0/849005692 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c080aafc0 con 0x7f3c080a8a20 2023-12-01T17:04:06.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.212+0000 7f3c07fff640 1 --2- 172.21.15.67:0/849005692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c080a8a20 0x7f3c080a8e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c080b8700 tx=0x7f3bf802b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcafff0d1a6777bc server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 1 -- 172.21.15.67:0/849005692 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8030860 con 0x7f3c080a8a20 2023-12-01T17:04:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: dump: 2023-12-01T17:04:06.220 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: _finish_auth 0 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 1 -- 172.21.15.67:0/849005692 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf80309c0 con 0x7f3c080a8a20 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:06.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 1 -- 172.21.15.67:0/849005692 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf8030d50 con 0x7f3c080a8a20 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c0e77c640 5 monclient: authenticate success, global_id 4357 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c0e77c640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c0e77c640 4 set_mon_vals no callback set 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.215+0000 7f3c06ffd640 10 monclient: dump: 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.222 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 10 monclient: shutdown 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 -- 172.21.15.67:0/849005692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c080a8a20 msgr2=0x7f3c080a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/849005692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c080a8a20 0x7f3c080a8e00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3c080b8700 tx=0x7f3bf802b9a0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 -- 172.21.15.67:0/849005692 shutdown_connections 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/849005692 >> v2:172.21.15.67:3300/0 conn(0x7f3c0813f800 0x7f3c08145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/849005692 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c080a8a20 0x7f3c080a8e00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 -- 172.21.15.67:0/849005692 >> 172.21.15.67:0/849005692 conn(0x7f3c080b7890 msgr2=0x7f3c080b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 -- 172.21.15.67:0/849005692 shutdown_connections 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.216+0000 7f3c0e77c640 1 -- 172.21.15.67:0/849005692 wait complete. 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 10 monclient: build_initial_monmap 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 10 monclient: monmap: 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:06.223 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 1 librados: starting msgr at 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 1 librados: starting objecter 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 1 Processor -- start 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 1 -- start start 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 1 librados: setting wanted keys 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 1 librados: calling monclient init 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.217+0000 7f3c0e77c640 10 monclient: init 2023-12-01T17:04:06.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c080a8a20 0x7f3c08137f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): picked mon.noname-a con 0x7f3c080a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c0813f800 0x7f3c081dab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): picked mon.noname-b con 0x7f3c0813f800 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3c08145ab0 0x7f3c08138710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): picked mon.noname-c con 0x7f3c08145ab0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c081da690 con 0x7f3c080a8a20 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08137900 con 0x7f3c0813f800 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c081db060 con 0x7f3c08145ab0 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:06.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0e77c640 10 monclient(hunting): authenticate will time out at 1377.456786s 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c07fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c080a8a20 0x7f3c08137f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c077fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c0813f800 0x7f3c081dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.218+0000 7f3c0cd19640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3c08145ab0 0x7f3c08138710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c0813f800 0x7f3c081dab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36718/0 (socket says 172.21.15.67:36718) 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 1 -- 172.21.15.67:0/3164549200 learned_addr learned my addr 172.21.15.67:0/3164549200 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 10 monclient(hunting): get_auth_request con 0x7f3c0813f800 auth_method 0 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c07fff640 10 monclient(hunting): get_auth_request con 0x7f3c080a8a20 auth_method 0 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c07fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c07fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c0cd19640 10 monclient(hunting): get_auth_request con 0x7f3c08145ab0 auth_method 0 2023-12-01T17:04:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c0cd19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c0cd19640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c0cd19640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c0cd19640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c0cd19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.219+0000 7f3c0cd19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 1 -- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3c08145ab0 msgr2=0x7f3c08138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3c08145ab0 0x7f3c08138710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 1 -- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c080a8a20 msgr2=0x7f3c08137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c080a8a20 0x7f3c08137f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 10 monclient: _finish_hunting 0 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 1 monclient: found mon.noname-b 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.222+0000 7f3c077fe640 1 -- 172.21.15.67:0/3164549200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf80094d0 con 0x7f3c0813f800 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.224+0000 7f3c077fe640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c0813f800 0x7f3c081dab20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c400 tx=0x7f3bf000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.224+0000 7f3c04ff9640 10 client.?.objecter ms_handle_connect 0x7f3c0813f800 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.224+0000 7f3c04ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c04ff9640 1 -- 172.21.15.67:0/3164549200 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf00052d0 con 0x7f3c0813f800 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c04ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c04ff9640 10 monclient: dump: 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c04ff9640 10 monclient: _finish_auth 0 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c04ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:06.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c0e77c640 5 monclient: authenticate success, global_id 4360 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c0e77c640 10 monclient: _renew_subs 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c0e77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.226+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c081db2e0 con 0x7f3c0813f800 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 10 monclient: _renew_subs 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c08138e30 con 0x7f3c0813f800 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 1 librados: init done 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.227+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c081da460 con 0x7f3c0813f800 2023-12-01T17:04:06.327 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-12-01T17:04:06.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-12-01T17:04:06.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.434+0000 7f3c04ff9640 1 -- 172.21.15.67:0/3164549200 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0005430 con 0x7f3c0813f800 2023-12-01T17:04:06.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.434+0000 7f3c04ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:06.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.434+0000 7f3c067fc640 4 set_mon_vals no callback set 2023-12-01T17:04:06.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.434+0000 7f37f7a7c640 10 monclient: get_monmap_and_config 2023-12-01T17:04:06.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.434+0000 7f37f7a7c640 10 monclient: build_initial_monmap 2023-12-01T17:04:06.436 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.434+0000 7f37f7a7c640 10 monclient: monmap: 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:06.436625+0000 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:06.436625+0000 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:06.437 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.435+0000 7f37f7a7c640 1 Processor -- start 2023-12-01T17:04:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.435+0000 7f37f7a7c640 1 -- start start 2023-12-01T17:04:06.438 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.435+0000 7f37f7a7c640 10 monclient: init 2023-12-01T17:04:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:06.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37f00a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37f01461c0 0x7f37f00a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f37f01461c0 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f37f013f0f0 0x7f37f013f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f37f013f0f0 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f0145620 con 0x7f37f00a8a20 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f00aace0 con 0x7f37f01461c0 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f00aafc0 con 0x7f37f013f0f0 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.436+0000 7f37f7a7c640 10 monclient(hunting): authenticate will time out at 1377.674853s 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.437+0000 7f37f5017640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37f01461c0 0x7f37f00a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.437+0000 7f37f5017640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f37f01461c0 0x7f37f00a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58102/0 (socket says 172.21.15.67:58102) 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.437+0000 7f37f5017640 1 -- 172.21.15.67:0/1952613244 learned_addr learned my addr 172.21.15.67:0/1952613244 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.437+0000 7f37f5017640 10 monclient(hunting): get_auth_request con 0x7f37f01461c0 auth_method 0 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.437+0000 7f37f5017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.437+0000 7f37f5017640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.437+0000 7f37f5017640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.438+0000 7f37f5017640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.438+0000 7f37f5017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.438+0000 7f37f5017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 10 monclient(hunting): handle_auth_done global_id 4317 payload 274 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 1 -- 172.21.15.67:0/1952613244 >> v2:172.21.15.92:3300/0 conn(0x7f37f013f0f0 msgr2=0x7f37f013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 1 --2- 172.21.15.67:0/1952613244 >> v2:172.21.15.92:3300/0 conn(0x7f37f013f0f0 0x7f37f013f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 1 -- 172.21.15.67:0/1952613244 >> v1:172.21.15.67:6789/0 conn(0x7f37f00a8a20 legacy=0x7f37f0145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 10 monclient: _finish_hunting 0 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 1 monclient: found mon.noname-a 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:06.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.440+0000 7f37f5017640 1 -- 172.21.15.67:0/1952613244 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f013fa90 con 0x7f37f01461c0 2023-12-01T17:04:06.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.444+0000 7f37f5017640 1 --2- 172.21.15.67:0/1952613244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f01461c0 0x7f37f00a8e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f37f00b3990 tx=0x7f37e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a1af72a586f960 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 1 -- 172.21.15.67:0/1952613244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4030860 con 0x7f37f01461c0 2023-12-01T17:04:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: dump: 2023-12-01T17:04:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.449 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: _finish_auth 0 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: _check_auth_tickets 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 1 -- 172.21.15.67:0/1952613244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e40309c0 con 0x7f37f01461c0 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 1 -- 172.21.15.67:0/1952613244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4030d50 con 0x7f37f01461c0 2023-12-01T17:04:06.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f7a7c640 5 monclient: authenticate success, global_id 4317 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f7a7c640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f7a7c640 4 set_mon_vals no callback set 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.448+0000 7f37f4816640 10 monclient: dump: 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.451 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 10 monclient: shutdown 2023-12-01T17:04:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 -- 172.21.15.67:0/1952613244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f01461c0 msgr2=0x7f37f00a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/1952613244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f01461c0 0x7f37f00a8e00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f37f00b3990 tx=0x7f37e402b9a0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 -- 172.21.15.67:0/1952613244 shutdown_connections 2023-12-01T17:04:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/1952613244 >> v2:172.21.15.92:3300/0 conn(0x7f37f013f0f0 0x7f37f013f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.454 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/1952613244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f01461c0 0x7f37f00a8e00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 -- 172.21.15.67:0/1952613244 >> 172.21.15.67:0/1952613244 conn(0x7f37f00b7890 msgr2=0x7f37f00b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 -- 172.21.15.67:0/1952613244 shutdown_connections 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 1 -- 172.21.15.67:0/1952613244 wait complete. 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 10 monclient: build_initial_monmap 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.449+0000 7f37f7a7c640 10 monclient: monmap: 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:06.455 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 librados: starting msgr at 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 librados: starting objecter 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 Processor -- start 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 -- start start 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 librados: setting wanted keys 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 librados: calling monclient init 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 10 monclient: init 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f37f00a8a20 0x7f37f0144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 10 monclient(hunting): picked mon.noname-c con 0x7f37f00a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f013f0f0 0x7f37f01d70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 10 monclient(hunting): picked mon.noname-a con 0x7f37f013f0f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:06.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.450+0000 7f37f7a7c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f01461c0 0x7f37f01d75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 10 monclient(hunting): picked mon.noname-b con 0x7f37f01461c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f00b7220 con 0x7f37f013f0f0 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f0144410 con 0x7f37f01461c0 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f01446f0 con 0x7f37f00a8a20 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f7a7c640 10 monclient(hunting): authenticate will time out at 1377.689427s 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5818640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f37f00a8a20 0x7f37f0144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5017640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f013f0f0 0x7f37f01d70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5818640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f37f00a8a20 0x7f37f0144c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:42006/0 (socket says 172.21.15.67:42006) 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5818640 1 -- 172.21.15.67:0/3959312137 learned_addr learned my addr 172.21.15.67:0/3959312137 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:06.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5818640 10 monclient(hunting): get_auth_request con 0x7f37f00a8a20 auth_method 0 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5818640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5818640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.451+0000 7f37f5818640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5017640 10 monclient(hunting): get_auth_request con 0x7f37f013f0f0 auth_method 0 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5017640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5017640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5017640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5818640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5818640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5818640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5017640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5017640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.452+0000 7f37f5017640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 10 monclient(hunting): handle_auth_done global_id 4298 payload 824 2023-12-01T17:04:06.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 1 -- 172.21.15.67:0/3959312137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f01461c0 msgr2=0x7f37f01d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f01461c0 0x7f37f01d75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 1 -- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f013f0f0 msgr2=0x7f37f01d70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f013f0f0 0x7f37f01d70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 10 monclient: _finish_hunting 0 2023-12-01T17:04:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 1 monclient: found mon.noname-c 2023-12-01T17:04:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.455+0000 7f37f5818640 1 -- 172.21.15.67:0/3959312137 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e40094d0 con 0x7f37f00a8a20 2023-12-01T17:04:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.458+0000 7f37f5818640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f37f00a8a20 0x7f37f0144c10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00c430 tx=0x7f37ec00c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.458+0000 7f37e27fc640 10 client.?.objecter ms_handle_connect 0x7f37f00a8a20 2023-12-01T17:04:06.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.458+0000 7f37e27fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 1 -- 172.21.15.67:0/3959312137 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0052d0 con 0x7f37f00a8a20 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: dump: 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.464 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 1 -- 172.21.15.67:0/3959312137 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37ec005450 con 0x7f37f00a8a20 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 1 -- 172.21.15.67:0/3959312137 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec007260 con 0x7f37f00a8a20 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e27fc640 10 monclient: dump: 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.465 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 5 monclient: authenticate success, global_id 4298 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 10 monclient: _renew_subs 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37e3fff640 4 set_mon_vals no callback set 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f01d7b30 con 0x7f37f00a8a20 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 10 monclient: _renew_subs 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f0145330 con 0x7f37f00a8a20 2023-12-01T17:04:06.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.462+0000 7f37f7a7c640 1 librados: init done 2023-12-01T17:04:06.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.463+0000 7f37f7a7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.463+0000 7f37f7a7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.463+0000 7f37f7a7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:06.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.463+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f01d7d20 con 0x7f37f00a8a20 2023-12-01T17:04:06.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.470+0000 7f3c04ff9640 1 -- 172.21.15.67:0/3164549200 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bf0007220 con 0x7f3c0813f800 2023-12-01T17:04:06.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.470+0000 7f3c04ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.470+0000 7f3c04ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:06.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.470+0000 7f3c04ff9640 10 monclient: dump: 2023-12-01T17:04:06.471 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.471 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.471 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.472 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.472 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.472 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.472 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.472 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.472 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.472 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.471+0000 7f3c04ff9640 1 -- 172.21.15.67:0/3164549200 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3bf0007440 con 0x7f3c0813f800 2023-12-01T17:04:06.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.471+0000 7f3c04ff9640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3be806c820 0x7f3be806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.471+0000 7f3c07fff640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3be806c820 0x7f3be806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.471+0000 7f3c07fff640 10 monclient: get_auth_request con 0x7f3be806c820 auth_method 0 2023-12-01T17:04:06.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.472+0000 7f3c04ff9640 10 client.4360.objecter ms_handle_connect 0x7f3be806c820 2023-12-01T17:04:06.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.472+0000 7f3c07fff640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3be806c820 0x7f3be806ece0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8036000 tx=0x7f3bf8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.474+0000 7f3c04ff9640 1 -- 172.21.15.67:0/3164549200 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3bf0057b60 con 0x7f3c0813f800 2023-12-01T17:04:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.474+0000 7f3c04ff9640 10 client.4360.objecter ms_dispatch 0x7f3c08000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.474+0000 7f3c04ff9640 3 client.4360.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.474+0000 7f3c04ff9640 3 client.4360.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:06.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.474+0000 7f3c04ff9640 20 client.4360.objecter dump_active .. 0 homeless 2023-12-01T17:04:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.484+0000 7f3c04ff9640 1 -- 172.21.15.67:0/3164549200 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bf0053ea0 con 0x7f3c0813f800 2023-12-01T17:04:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.484+0000 7f3c04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.484+0000 7f3c04ff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:06.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.484+0000 7f37e27fc640 1 -- 172.21.15.67:0/3959312137 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f37ec0073c0 con 0x7f37f00a8a20 2023-12-01T17:04:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.485+0000 7f37e27fc640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f37d406c870 0x7f37d406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.485+0000 7f37f5017640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f37d406c870 0x7f37d406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.485+0000 7f37f5017640 10 monclient: get_auth_request con 0x7f37d406c870 auth_method 0 2023-12-01T17:04:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.485+0000 7f37e27fc640 10 client.4298.objecter ms_handle_connect 0x7f37d406c870 2023-12-01T17:04:06.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.485+0000 7f37f5017640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f37d406c870 0x7f37d406ed30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37e4036000 tx=0x7f37e4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.489+0000 7f37e27fc640 1 -- 172.21.15.67:0/3959312137 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f37ec057cc0 con 0x7f37f00a8a20 2023-12-01T17:04:06.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.489+0000 7f37e27fc640 10 client.4298.objecter ms_dispatch 0x7f37f0000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:06.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.489+0000 7f37e27fc640 3 client.4298.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:06.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.489+0000 7f37e27fc640 3 client.4298.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:06.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.489+0000 7f37e27fc640 20 client.4298.objecter dump_active .. 0 homeless 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.499+0000 7fc6a8c34640 10 monclient: get_monmap_and_config 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.499+0000 7fc6a8c34640 10 monclient: build_initial_monmap 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.499+0000 7fc6a8c34640 10 monclient: monmap: 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:06.500932+0000 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:06.500932+0000 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:06.500 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:06.501 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:06.501 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:06.501 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:06.501 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.499+0000 7fc6a8c34640 1 Processor -- start 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.499+0000 7fc6a8c34640 1 -- start start 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.499+0000 7fc6a8c34640 10 monclient: init 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc6a40a8a20 0x7fc6a40a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a40a8a20 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6a4145ab0 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6a413f800 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a4145620 con 0x7fc6a413f800 2023-12-01T17:04:06.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a40aace0 con 0x7fc6a4145ab0 2023-12-01T17:04:06.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a40aafc0 con 0x7fc6a40a8a20 2023-12-01T17:04:06.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:06.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.500+0000 7fc6a8c34640 10 monclient(hunting): authenticate will time out at 1377.739119s 2023-12-01T17:04:06.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.501+0000 7fc6a259c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc6a40a8a20 0x7fc6a40a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.501+0000 7fc6a259c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc6a40a8a20 0x7fc6a40a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36748/0 (socket says 172.21.15.67:36748) 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.501+0000 7fc6a259c640 1 -- 172.21.15.67:0/1680232720 learned_addr learned my addr 172.21.15.67:0/1680232720 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.501+0000 7fc6a259c640 10 monclient(hunting): get_auth_request con 0x7fc6a40a8a20 auth_method 0 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.501+0000 7fc6a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.501+0000 7fc6a259c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.501+0000 7fc6a259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.502+0000 7fc6a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.502+0000 7fc6a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.502+0000 7fc6a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 10 monclient(hunting): handle_auth_done global_id 4363 payload 274 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 1 -- 172.21.15.67:0/1680232720 >> v1:172.21.15.92:6789/0 conn(0x7fc6a4145ab0 legacy=0x7fc6a413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 1 -- 172.21.15.67:0/1680232720 >> v1:172.21.15.67:6789/0 conn(0x7fc6a413f800 legacy=0x7fc6a414c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 10 monclient: _finish_hunting 0 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 1 monclient: found mon.noname-c 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.504+0000 7fc6a259c640 1 -- 172.21.15.67:0/1680232720 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a40b2210 con 0x7fc6a40a8a20 2023-12-01T17:04:06.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.508+0000 7fc6a259c640 1 --2- 172.21.15.67:0/1680232720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a40a8a20 0x7fc6a40a8e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc694009870 tx=0x7fc69402ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9f015d93a97bb5a server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.512+0000 7fc6a159a640 1 -- 172.21.15.67:0/1680232720 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69402f840 con 0x7fc6a40a8a20 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.512+0000 7fc6a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.512+0000 7fc6a159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.512+0000 7fc6a159a640 10 monclient: dump: 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.514 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 10 monclient: _finish_auth 0 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 10 monclient: _check_auth_tickets 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 1 -- 172.21.15.67:0/1680232720 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69402f9a0 con 0x7fc6a40a8a20 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 1 -- 172.21.15.67:0/1680232720 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69402fd30 con 0x7fc6a40a8a20 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a159a640 10 monclient: dump: 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.515 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a8c34640 5 monclient: authenticate success, global_id 4363 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a8c34640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:06.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a8c34640 4 set_mon_vals no callback set 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a8c34640 10 monclient: shutdown 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a8c34640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a8c34640 1 -- 172.21.15.67:0/1680232720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a40a8a20 msgr2=0x7fc6a40a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.513+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/1680232720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a40a8a20 0x7fc6a40a8e00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc694009870 tx=0x7fc69402ba20 comp rx=0 tx=0).stop 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.514+0000 7fc6a8c34640 1 -- 172.21.15.67:0/1680232720 shutdown_connections 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.514+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/1680232720 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a40a8a20 0x7fc6a40a8e00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.514+0000 7fc6a8c34640 1 -- 172.21.15.67:0/1680232720 >> 172.21.15.67:0/1680232720 conn(0x7fc6a40b7890 msgr2=0x7fc6a40b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.514+0000 7fc6a8c34640 1 -- 172.21.15.67:0/1680232720 shutdown_connections 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.514+0000 7fc6a8c34640 1 -- 172.21.15.67:0/1680232720 wait complete. 2023-12-01T17:04:06.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 10 monclient: build_initial_monmap 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 10 monclient: monmap: 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 1 librados: starting msgr at 2023-12-01T17:04:06.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 1 librados: starting objecter 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 1 Processor -- start 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 1 -- start start 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 1 librados: setting wanted keys 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 1 librados: calling monclient init 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.515+0000 7fc6a8c34640 10 monclient: init 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a40a8a20 0x7fc6a41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): picked mon.noname-a con 0x7fc6a40a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6a413f800 0x7fc6a41413b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): picked mon.noname-c con 0x7fc6a413f800 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a4145ab0 0x7fc6a40b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): picked mon.noname-b con 0x7fc6a4145ab0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:06.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a41366d0 con 0x7fc6a40a8a20 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a41429b0 con 0x7fc6a4145ab0 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a4142c20 con 0x7fc6a413f800 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.516+0000 7fc6a8c34640 10 monclient(hunting): authenticate will time out at 1377.755268s 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a40a8a20 0x7fc6a41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a2d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a4145ab0 0x7fc6a40b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a259c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a40a8a20 0x7fc6a41407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58120/0 (socket says 172.21.15.67:58120) 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a259c640 1 -- 172.21.15.67:0/2677968819 learned_addr learned my addr 172.21.15.67:0/2677968819 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a259c640 10 monclient(hunting): get_auth_request con 0x7fc6a40a8a20 auth_method 0 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a259c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.517+0000 7fc6a259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a2d9d640 10 monclient(hunting): get_auth_request con 0x7fc6a4145ab0 auth_method 0 2023-12-01T17:04:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a2d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:06.523 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.518+0000 7fc6a259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 10 monclient(hunting): handle_auth_done global_id 4369 payload 824 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 1 -- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6a413f800 msgr2=0x7fc6a41413b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.524 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6a413f800 0x7fc6a41413b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 1 -- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a40a8a20 msgr2=0x7fc6a41407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a40a8a20 0x7fc6a41407c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 1 monclient: found mon.noname-b 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.521+0000 7fc6a2d9d640 1 -- 172.21.15.67:0/2677968819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6940094d0 con 0x7fc6a4145ab0 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.523+0000 7fc6a2d9d640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a4145ab0 0x7fc6a40b05b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c690 tx=0x7fc69c00cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.523+0000 7fc692ffd640 10 client.?.objecter ms_handle_connect 0x7fc6a4145ab0 2023-12-01T17:04:06.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.523+0000 7fc692ffd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 1 -- 172.21.15.67:0/2677968819 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c003b70 con 0x7fc6a4145ab0 2023-12-01T17:04:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: dump: 2023-12-01T17:04:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.528 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: _finish_auth 0 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 1 -- 172.21.15.67:0/2677968819 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc69c003cd0 con 0x7fc6a4145ab0 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 5 monclient: authenticate success, global_id 4369 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 10 monclient: _renew_subs 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a4141a20 con 0x7fc6a4145ab0 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 1 -- 172.21.15.67:0/2677968819 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc69c005310 con 0x7fc6a4145ab0 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc692ffd640 10 monclient: dump: 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:06.530 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:06.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:06.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 10 monclient: _renew_subs 2023-12-01T17:04:06.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a40b0af0 con 0x7fc6a4145ab0 2023-12-01T17:04:06.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a8c34640 1 librados: init done 2023-12-01T17:04:06.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.526+0000 7fc6a0d99640 4 set_mon_vals no callback set 2023-12-01T17:04:06.531 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.528+0000 7fc6a8c34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.528+0000 7fc6a8c34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.528+0000 7fc6a8c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.528+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a404e840 con 0x7fc6a4145ab0 2023-12-01T17:04:06.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.545+0000 7fc692ffd640 1 -- 172.21.15.67:0/2677968819 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc69c005470 con 0x7fc6a4145ab0 2023-12-01T17:04:06.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.545+0000 7fc692ffd640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc68406c870 0x7fc68406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:06.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.546+0000 7fc6a259c640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc68406c870 0x7fc68406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:06.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.546+0000 7fc6a259c640 10 monclient: get_auth_request con 0x7fc68406c870 auth_method 0 2023-12-01T17:04:06.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.546+0000 7fc692ffd640 10 client.4369.objecter ms_handle_connect 0x7fc68406c870 2023-12-01T17:04:06.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.546+0000 7fc6a259c640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc68406c870 0x7fc68406ed30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc694009870 tx=0x7fc694008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:06.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.547+0000 7fc692ffd640 1 -- 172.21.15.67:0/2677968819 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc69c059230 con 0x7fc6a4145ab0 2023-12-01T17:04:06.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.547+0000 7fc692ffd640 10 client.4369.objecter ms_dispatch 0x7fc6a4000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.547+0000 7fc692ffd640 3 client.4369.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.547+0000 7fc692ffd640 3 client.4369.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:06.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.547+0000 7fc692ffd640 20 client.4369.objecter dump_active .. 0 homeless 2023-12-01T17:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.671+0000 7f3c0e77c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-01T17:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.671+0000 7f3c0e77c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-01T17:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.671+0000 7f3c0e77c640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.671+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f3c081da460 con 0x7f3c0813f800 2023-12-01T17:04:06.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.764+0000 7f37e27fc640 1 -- 172.21.15.67:0/3959312137 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f37ec054000 con 0x7f37f00a8a20 2023-12-01T17:04:06.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.764+0000 7f37e27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.764+0000 7f37e27fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:06.775 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.774+0000 7fc692ffd640 1 -- 172.21.15.67:0/2677968819 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc69c054e70 con 0x7fc6a4145ab0 2023-12-01T17:04:06.775 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.774+0000 7fc692ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:06.775 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.774+0000 7fc692ffd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.780+0000 7f3c04ff9640 1 -- 172.21.15.67:0/3164549200 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3c081da460 con 0x7f3c0813f800 2023-12-01T17:04:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.780+0000 7f3c04ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-12-01T17:04:06.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.780+0000 7f3c04ff9640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.782 INFO:teuthology.orchestra.run.smithi067.stdout:60129542148 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 10 librados: watch_flush enter 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 10 librados: watch_flush exit 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 20 client.4360.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 10 client.4360.objecter successfully canceled tick 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3be806c820 msgr2=0x7f3be806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3be806c820 0x7f3be806ece0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3bf8036000 tx=0x7f3bf8008030 comp rx=0 tx=0).stop 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 10 monclient: shutdown 2023-12-01T17:04:06.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c0813f800 msgr2=0x7f3c081dab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c0813f800 0x7f3c081dab20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3bf000c400 tx=0x7f3bf000c8c0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 shutdown_connections 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3be806c820 0x7f3be806ece0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f3c08145ab0 0x7f3c08138710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3c0813f800 0x7f3c081dab20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 --2- 172.21.15.67:0/3164549200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c080a8a20 0x7f3c08137f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.783+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 >> 172.21.15.67:0/3164549200 conn(0x7f3c080b7890 msgr2=0x7f3c0813b790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.784+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 shutdown_connections 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.784+0000 7f3c0e77c640 1 -- 172.21.15.67:0/3164549200 wait complete. 2023-12-01T17:04:06.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.784+0000 7f3c0e77c640 1 librados: shutdown 2023-12-01T17:04:06.800 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542148 for osd.5 2023-12-01T17:04:06.801 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:06.954 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.952+0000 7f37f7a7c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-01T17:04:06.954 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.952+0000 7f37f7a7c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-01T17:04:06.954 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.952+0000 7f37f7a7c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:06.954 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.952+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f37f004e840 con 0x7f37f00a8a20 2023-12-01T17:04:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.966+0000 7fc6a8c34640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-01T17:04:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.966+0000 7fc6a8c34640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-01T17:04:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.966+0000 7fc6a8c34640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:06.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.966+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc6a40008d0 con 0x7fc6a4145ab0 2023-12-01T17:04:06.971 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.970+0000 7fc692ffd640 1 -- 172.21.15.67:0/2677968819 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc69c059e60 con 0x7fc6a4145ab0 2023-12-01T17:04:06.971 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.970+0000 7fc692ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-12-01T17:04:06.971 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.970+0000 7fc692ffd640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.971 INFO:teuthology.orchestra.run.smithi067.stdout:68719476743 2023-12-01T17:04:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 10 librados: watch_flush enter 2023-12-01T17:04:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 10 librados: watch_flush exit 2023-12-01T17:04:06.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 20 client.4369.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 10 client.4369.objecter successfully canceled tick 2023-12-01T17:04:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc68406c870 msgr2=0x7fc68406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc68406c870 0x7fc68406ed30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc694009870 tx=0x7fc694008030 comp rx=0 tx=0).stop 2023-12-01T17:04:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 10 monclient: shutdown 2023-12-01T17:04:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a4145ab0 msgr2=0x7fc6a40b05b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.975 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.972+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a4145ab0 0x7fc6a40b05b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc69c00c690 tx=0x7fc69c00cb50 comp rx=0 tx=0).stop 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 shutdown_connections 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc6a4145ab0 0x7fc6a40b05b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc68406c870 0x7fc68406ed30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc6a413f800 0x7fc6a41413b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 --2- 172.21.15.67:0/2677968819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6a40a8a20 0x7fc6a41407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 >> 172.21.15.67:0/2677968819 conn(0x7fc6a40b7890 msgr2=0x7fc6a40b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 shutdown_connections 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 -- 172.21.15.67:0/2677968819 wait complete. 2023-12-01T17:04:06.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.973+0000 7fc6a8c34640 1 librados: shutdown 2023-12-01T17:04:06.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.974+0000 7f37e27fc640 1 -- 172.21.15.67:0/3959312137 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f37ec057620 con 0x7f37f00a8a20 2023-12-01T17:04:06.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.974+0000 7f37e27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-12-01T17:04:06.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.974+0000 7f37e27fc640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:06.977 INFO:teuthology.orchestra.run.smithi067.stdout:60129542149 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.976+0000 7f37f7a7c640 10 librados: watch_flush enter 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.976+0000 7f37f7a7c640 10 librados: watch_flush exit 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.976+0000 7f37f7a7c640 20 client.4298.objecter shutdown clearing up homeless session... 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.976+0000 7f37f7a7c640 10 client.4298.objecter successfully canceled tick 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f37d406c870 msgr2=0x7f37d406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f37d406c870 0x7f37d406ed30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f37e4036000 tx=0x7f37e4008030 comp rx=0 tx=0).stop 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 10 monclient: shutdown 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f37f00a8a20 msgr2=0x7f37f0144c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f37f00a8a20 0x7f37f0144c10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37ec00c430 tx=0x7f37ec00c8f0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 shutdown_connections 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37f01461c0 0x7f37f01d75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f37f013f0f0 0x7f37f01d70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f37d406c870 0x7f37d406ed30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 --2- 172.21.15.67:0/3959312137 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f37f00a8a20 0x7f37f0144c10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 >> 172.21.15.67:0/3959312137 conn(0x7f37f00b7890 msgr2=0x7f37f00b49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 shutdown_connections 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 -- 172.21.15.67:0/3959312137 wait complete. 2023-12-01T17:04:06.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:06.977+0000 7f37f7a7c640 1 librados: shutdown 2023-12-01T17:04:06.987 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.3 2023-12-01T17:04:06.987 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:06.991 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542148 got 60129542149 for osd.6 2023-12-01T17:04:06.992 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:07.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-12-01T17:04:07.044 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-12-01T17:04:07.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-12-01T17:04:07.060 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-12-01T17:04:07.063 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.181+0000 7f57a513e640 10 monclient: get_monmap_and_config 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.181+0000 7f57a513e640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.181+0000 7f57a513e640 10 monclient: monmap: 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:07.183591+0000 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:07.183591+0000 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:07.183 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:07.184 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.182+0000 7f57a513e640 1 Processor -- start 2023-12-01T17:04:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.183+0000 7f57a513e640 1 -- start start 2023-12-01T17:04:07.184 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.183+0000 7f57a513e640 10 monclient: init 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57a00a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57a01461c0 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f57a00a8e00 0x7f57a013f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient(hunting): picked mon.noname-c con 0x7f57a00a8e00 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a0145620 con 0x7f57a00a8a20 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a00aace0 con 0x7f57a01461c0 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a00aafc0 con 0x7f57a00a8e00 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.186 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.184+0000 7f57a513e640 10 monclient(hunting): authenticate will time out at 1378.422951s 2023-12-01T17:04:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.185+0000 7f579f59e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f57a00a8e00 0x7f57a013f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.185+0000 7f579ed9d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f57a00a8a20 0x7f57a0145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57608/0 (socket says 172.21.15.67:57608) 2023-12-01T17:04:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.185+0000 7f579ed9d640 1 -- 172.21.15.67:0/2999840929 learned_addr learned my addr 172.21.15.67:0/2999840929 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.185+0000 7f579f59e640 10 monclient(hunting): get_auth_request con 0x7f57a00a8e00 auth_method 0 2023-12-01T17:04:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.185+0000 7f579f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.185+0000 7f579f59e640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.187 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.185+0000 7f579f59e640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.186+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.186+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.186+0000 7f579f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.187+0000 7f579f59e640 10 monclient(hunting): handle_auth_done global_id 4372 payload 274 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.187+0000 7f579f59e640 1 -- 172.21.15.67:0/2999840929 >> v1:172.21.15.92:6789/0 conn(0x7f57a01461c0 legacy=0x7f57a013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.188+0000 7f579f59e640 1 -- 172.21.15.67:0/2999840929 >> v1:172.21.15.67:6789/0 conn(0x7f57a00a8a20 legacy=0x7f57a0145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.188+0000 7f579f59e640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.188+0000 7f579f59e640 1 monclient: found mon.noname-c 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.188+0000 7f579f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.188+0000 7f579f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.188+0000 7f579f59e640 1 -- 172.21.15.67:0/2999840929 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a00b2210 con 0x7f57a00a8e00 2023-12-01T17:04:07.192 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.191+0000 7f579f59e640 1 --2- 172.21.15.67:0/2999840929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a00a8e00 0x7f57a013f800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5798009d90 tx=0x7f579802bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5558bbbe93ba1eb5 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.196 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 1 -- 172.21.15.67:0/2999840929 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579802ccf0 con 0x7f57a00a8e00 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: dump: 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.197 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 1 -- 172.21.15.67:0/2999840929 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f579802ce50 con 0x7f57a00a8e00 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 1 -- 172.21.15.67:0/2999840929 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5798034660 con 0x7f57a00a8e00 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f579e59c640 10 monclient: dump: 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.198 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f57a513e640 5 monclient: authenticate success, global_id 4372 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f57a513e640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:07.199 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.195+0000 7f57a513e640 4 set_mon_vals no callback set 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 10 monclient: shutdown 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 1 -- 172.21.15.67:0/2999840929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a00a8e00 msgr2=0x7f57a013f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 1 --2- 172.21.15.67:0/2999840929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a00a8e00 0x7f57a013f800 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5798009d90 tx=0x7f579802bf40 comp rx=0 tx=0).stop 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 1 -- 172.21.15.67:0/2999840929 shutdown_connections 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 1 --2- 172.21.15.67:0/2999840929 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a00a8e00 0x7f57a013f800 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 1 -- 172.21.15.67:0/2999840929 >> 172.21.15.67:0/2999840929 conn(0x7f57a00b7890 msgr2=0x7f57a00b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 1 -- 172.21.15.67:0/2999840929 shutdown_connections 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.196+0000 7f57a513e640 1 -- 172.21.15.67:0/2999840929 wait complete. 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.197+0000 7f57a513e640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.198+0000 7f57a513e640 10 monclient: monmap: 2023-12-01T17:04:07.201 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.198+0000 7f57a513e640 1 librados: starting msgr at 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.198+0000 7f57a513e640 1 librados: starting objecter 2023-12-01T17:04:07.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.198+0000 7f57a513e640 1 Processor -- start 2023-12-01T17:04:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.198+0000 7fd636ec9640 10 monclient: get_monmap_and_config 2023-12-01T17:04:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.198+0000 7fd636ec9640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.203 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.198+0000 7fd636ec9640 10 monclient: monmap: 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:07.200648+0000 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:07.200648+0000 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:07.204 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7f57a513e640 1 -- start start 2023-12-01T17:04:07.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7f57a513e640 1 librados: setting wanted keys 2023-12-01T17:04:07.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7f57a513e640 1 librados: calling monclient init 2023-12-01T17:04:07.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7f57a513e640 10 monclient: init 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7fbcf56e5640 10 monclient: get_monmap_and_config 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7fbcf56e5640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7fbcf56e5640 10 monclient: monmap: 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:07.201057+0000 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:07.201057+0000 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:07.206 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:07.207 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7fbcf56e5640 1 Processor -- start 2023-12-01T17:04:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.199+0000 7fd636ec9640 1 Processor -- start 2023-12-01T17:04:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 1 -- start start 2023-12-01T17:04:07.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient: init 2023-12-01T17:04:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7f57a513e640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fd636ec9640 1 -- start start 2023-12-01T17:04:07.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fd636ec9640 10 monclient: init 2023-12-01T17:04:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7f57a513e640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:04:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7f57a513e640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f57a00a8a20 0x7f57a0134830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7f57a513e640 10 monclient(hunting): picked mon.noname-c con 0x7f57a00a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7f57a513e640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a8e00 0x7f57a0142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-12-01T17:04:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fbcf00a8a20 0x7fbcf00a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf00a8a20 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbcf0145ab0 0x7fbcf0145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf0145ab0 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:07.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 0x7fbcf013f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fd636ec9640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fd636ec9640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf013f0f0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf0146430 con 0x7fbcf00a8a20 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcf0145620 con 0x7fbcf0145ab0 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf00aace0 con 0x7fbcf013f0f0 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.200+0000 7fbcf56e5640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fbcf56e5640 10 monclient(hunting): authenticate will time out at 1378.439407s 2023-12-01T17:04:07.213 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6300a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f8f43a61640 10 monclient: get_monmap_and_config 2023-12-01T17:04:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f8f43a61640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6301461c0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f8f43a61640 10 monclient: monmap: 2023-12-01T17:04:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.214 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:07.203016+0000 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:07.203016+0000 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:07.215 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd6300a8e00 0x7fd63013f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fd6300a8e00 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd630145620 con 0x7fd6301461c0 2023-12-01T17:04:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 10 monclient(hunting): picked mon.noname-a con 0x7f57a00a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a01461c0 0x7f57a013e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6300aace0 con 0x7fd6300a8a20 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 10 monclient(hunting): picked mon.noname-b con 0x7f57a01461c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0135a00 con 0x7f57a00a8e00 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0142540 con 0x7f57a01461c0 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a0138410 con 0x7f57a00a8a20 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f57a513e640 10 monclient(hunting): authenticate will time out at 1378.439649s 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f5797fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a8e00 0x7f57a0142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f579ed9d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f57a00a8a20 0x7f57a0134830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6300aafc0 con 0x7fd6300a8e00 2023-12-01T17:04:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd636ec9640 10 monclient(hunting): authenticate will time out at 1378.440042s 2023-12-01T17:04:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fbcee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbcf0145ab0 0x7fbcf0145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f579ed9d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f57a00a8a20 0x7f57a0134830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:42032/0 (socket says 172.21.15.67:42032) 2023-12-01T17:04:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f579ed9d640 1 -- 172.21.15.67:0/910698061 learned_addr learned my addr 172.21.15.67:0/910698061 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f579ed9d640 10 monclient(hunting): get_auth_request con 0x7f57a00a8a20 auth_method 0 2023-12-01T17:04:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f579ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f579ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7f579ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd634c65640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fd6300a8a20 0x7fd630145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57848/0 (socket says 172.21.15.67:57848) 2023-12-01T17:04:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.201+0000 7fd634c65640 1 -- 172.21.15.67:0/535411133 learned_addr learned my addr 172.21.15.67:0/535411133 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f5797fff640 10 monclient(hunting): get_auth_request con 0x7f57a00a8e00 auth_method 0 2023-12-01T17:04:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f5797fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f5797fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f5797fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbcf0145ab0 0x7fbcf0145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36774/0 (socket says 172.21.15.67:36774) 2023-12-01T17:04:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 1 -- 172.21.15.67:0/1405442270 learned_addr learned my addr 172.21.15.67:0/1405442270 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 10 monclient(hunting): get_auth_request con 0x7fbcf0145ab0 auth_method 0 2023-12-01T17:04:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f8f43a61640 1 Processor -- start 2023-12-01T17:04:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f579ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f579ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f579ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f8f43a61640 1 -- start start 2023-12-01T17:04:07.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7f8f43a61640 10 monclient: init 2023-12-01T17:04:07.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.202+0000 7fbcee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f3c0a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8f3c1461c0 0x7f8f3c0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c1461c0 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f3c13f0f0 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c145620 con 0x7f8f3c0a8a20 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3c0aace0 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f3c13fc60 con 0x7f8f3c1461c0 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f8f43a61640 10 monclient(hunting): authenticate will time out at 1378.441540s 2023-12-01T17:04:07.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f5797fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f5797fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.203+0000 7f5797fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 10 monclient(hunting): handle_auth_done global_id 4375 payload 274 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 1 -- 172.21.15.67:0/1405442270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 msgr2=0x7fbcf013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 1 --2- 172.21.15.67:0/1405442270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 0x7fbcf013f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 1 -- 172.21.15.67:0/1405442270 >> v2:172.21.15.67:3300/0 conn(0x7fbcf00a8a20 msgr2=0x7fbcf00a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 1 --2- 172.21.15.67:0/1405442270 >> v2:172.21.15.67:3300/0 conn(0x7fbcf00a8a20 0x7fbcf00a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 1 monclient: found mon.noname-c 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.204+0000 7fbcee7fc640 1 -- 172.21.15.67:0/1405442270 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf00aafc0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 10 monclient(hunting): handle_auth_done global_id 4301 payload 824 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 1 -- 172.21.15.67:0/910698061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a01461c0 msgr2=0x7f57a013e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a01461c0 0x7f57a013e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 1 -- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a8e00 msgr2=0x7f57a0142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a8e00 0x7f57a0142a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 1 monclient: found mon.noname-c 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.227 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.205+0000 7f579ed9d640 1 -- 172.21.15.67:0/910698061 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5798009a70 con 0x7f57a00a8a20 2023-12-01T17:04:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.207+0000 7fd62f7fe640 1 -- 172.21.15.67:0/535411133 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174790807 0 0) 0x7fd6300aace0 con 0x7fd6300a8a20 2023-12-01T17:04:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.207+0000 7fd62f7fe640 10 monclient(hunting): _negotiate 2023-12-01T17:04:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.207+0000 7fd62f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.207+0000 7fd62f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.207+0000 7fd62f7fe640 10 monclient(hunting): my global_id is 4378 2023-12-01T17:04:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.207+0000 7fd62f7fe640 1 -- 172.21.15.67:0/535411133 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd620003520 con 0x7fd6300a8a20 2023-12-01T17:04:07.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.208+0000 7f579ed9d640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f57a00a8a20 0x7f57a0134830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f579000c4a0 tx=0x7f579000c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.208+0000 7f57977fe640 10 client.?.objecter ms_handle_connect 0x7f57a00a8a20 2023-12-01T17:04:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.208+0000 7f57977fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.210+0000 7fd635466640 1 --2- 172.21.15.67:0/535411133 >> v2:172.21.15.67:3300/0 conn(0x7fd6300a8e00 0x7fd63013f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.210+0000 7f8f41ffe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8f3c13f0f0 0x7f8f3c13f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57858/0 (socket says 172.21.15.67:57858) 2023-12-01T17:04:07.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.210+0000 7f8f41ffe640 1 -- 172.21.15.67:0/963702033 learned_addr learned my addr 172.21.15.67:0/963702033 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:07.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.211+0000 7f8f40ffc640 1 --2- 172.21.15.67:0/963702033 >> v2:172.21.15.92:3300/0 conn(0x7f8f3c1461c0 0x7f8f3c0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7fd635466640 10 monclient(hunting): get_auth_request con 0x7fd6300a8e00 auth_method 0 2023-12-01T17:04:07.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7fd635466640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7fd635466640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7f8f40ffc640 10 monclient(hunting): get_auth_request con 0x7f8f3c1461c0 auth_method 0 2023-12-01T17:04:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7f8f40ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7f8f40ffc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.231 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7f8f40ffc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7fd635466640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.212+0000 7f57977fe640 1 -- 172.21.15.67:0/910698061 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5790005ce0 con 0x7f57a00a8a20 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: dump: 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.233 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 1 -- 172.21.15.67:0/910698061 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5790005e40 con 0x7f57a00a8a20 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 1 -- 172.21.15.67:0/910698061 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57900072d0 con 0x7f57a00a8a20 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57977fe640 10 monclient: dump: 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.234 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 5 monclient: authenticate success, global_id 4301 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 10 monclient: _renew_subs 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a01385d0 con 0x7f57a00a8a20 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f579dd9b640 4 set_mon_vals no callback set 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 10 monclient: _renew_subs 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a01d70a0 con 0x7f57a00a8a20 2023-12-01T17:04:07.235 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f57a513e640 1 librados: init done 2023-12-01T17:04:07.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7fd635466640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7fd635466640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7fd635466640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.236 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.213+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.214+0000 7f57a513e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.214+0000 7f57a513e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.214+0000 7f57a513e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.214+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a004e840 con 0x7f57a00a8a20 2023-12-01T17:04:07.237 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2023-12-01T17:04:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 1 -- 172.21.15.67:0/535411133 >> v1:172.21.15.92:6789/0 conn(0x7fd6300a8a20 legacy=0x7fd630145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 1 -- 172.21.15.67:0/535411133 >> v1:172.21.15.67:6789/0 conn(0x7fd6301461c0 legacy=0x7fd63013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 1 monclient: found mon.noname-a 2023-12-01T17:04:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:07.238 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7fd635466640 1 -- 172.21.15.67:0/535411133 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6300b2210 con 0x7fd6300a8e00 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_done global_id 4381 payload 274 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.215+0000 7f8f40ffc640 1 -- 172.21.15.67:0/963702033 >> v1:172.21.15.92:6789/0 conn(0x7f8f3c13f0f0 legacy=0x7f8f3c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.216+0000 7f8f40ffc640 1 -- 172.21.15.67:0/963702033 >> v1:172.21.15.67:6789/0 conn(0x7f8f3c0a8a20 legacy=0x7f8f3c145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.216+0000 7f8f40ffc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.216+0000 7f8f40ffc640 1 monclient: found mon.noname-c 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.216+0000 7f8f40ffc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.216+0000 7f8f40ffc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.216+0000 7f8f40ffc640 1 -- 172.21.15.67:0/963702033 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3c0aafc0 con 0x7f8f3c1461c0 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.218+0000 7f8f40ffc640 1 --2- 172.21.15.67:0/963702033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c1461c0 0x7f8f3c0a8e00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0097a0 tx=0x7f8f2c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ef2523b9222b6c server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.239 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.220+0000 7fd635466640 1 --2- 172.21.15.67:0/535411133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8e00 0x7fd63013f800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd628009d90 tx=0x7fd62802bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437312cb61e29b76 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.240 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.222+0000 7fbcee7fc640 1 --2- 172.21.15.67:0/1405442270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0145ef0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbcf00a9a50 tx=0x7fbce402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ac770274c0d0c3a server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.222+0000 7fc3edffc640 10 monclient: get_monmap_and_config 2023-12-01T17:04:07.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.222+0000 7fc3edffc640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.222+0000 7fc3edffc640 10 monclient: monmap: 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:07.223894+0000 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:07.223894+0000 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 1 Processor -- start 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 1 -- start start 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient: init 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3e80715f0 0x7fc3e813cf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80715f0 addr v2:172.21.15.67:3300/0 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3e80aea80 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc3e813fe30 0x7fc3e81402a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e813fe30 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3e80af6d0 con 0x7fc3e80aea80 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80a8880 con 0x7fc3e80715f0 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3e80719d0 con 0x7fc3e813fe30 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.243 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.223+0000 7fc3edffc640 10 monclient(hunting): authenticate will time out at 1378.462381s 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 1 -- 172.21.15.67:0/963702033 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c031830 con 0x7f8f3c1461c0 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: dump: 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.246 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 1 -- 172.21.15.67:0/963702033 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c031990 con 0x7f8f3c1461c0 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 1 -- 172.21.15.67:0/963702033 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c031d20 con 0x7f8f3c1461c0 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f33fff640 10 monclient: dump: 2023-12-01T17:04:07.247 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f43a61640 5 monclient: authenticate success, global_id 4381 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f43a61640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f43a61640 4 set_mon_vals no callback set 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.225+0000 7f8f43a61640 10 monclient: shutdown 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 1 -- 172.21.15.67:0/963702033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c1461c0 msgr2=0x7f8f3c0a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 1 --2- 172.21.15.67:0/963702033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c1461c0 0x7f8f3c0a8e00 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0097a0 tx=0x7f8f2c02b8d0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 1 -- 172.21.15.67:0/963702033 shutdown_connections 2023-12-01T17:04:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 1 --2- 172.21.15.67:0/963702033 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c1461c0 0x7f8f3c0a8e00 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 1 -- 172.21.15.67:0/963702033 >> 172.21.15.67:0/963702033 conn(0x7f8f3c0b7890 msgr2=0x7f8f3c0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 1 -- 172.21.15.67:0/963702033 shutdown_connections 2023-12-01T17:04:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 1 -- 172.21.15.67:0/963702033 wait complete. 2023-12-01T17:04:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 1 -- 172.21.15.67:0/535411133 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62802cd00 con 0x7fd6300a8e00 2023-12-01T17:04:07.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: dump: 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.250 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 1 -- 172.21.15.67:0/535411133 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62802ce60 con 0x7fd6300a8e00 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd636ec9640 5 monclient: authenticate success, global_id 4335 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd636ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd636ec9640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd636ec9640 4 set_mon_vals no callback set 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 1 -- 172.21.15.67:0/535411133 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628034660 con 0x7fd6300a8e00 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fd62f7fe640 10 monclient: dump: 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7fc3e77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3e80715f0 0x7fc3e813cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.226+0000 7f8f43a61640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 10 monclient: monmap: 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:07.253 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fc3e80715f0 0x7fc3e813cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58156/0 (socket says 172.21.15.67:58156) 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 1 -- 172.21.15.67:0/1727812470 learned_addr learned my addr 172.21.15.67:0/1727812470 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 10 monclient(hunting): get_auth_request con 0x7fc3e80715f0 auth_method 0 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 1 librados: starting msgr at 2023-12-01T17:04:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 1 librados: starting objecter 2023-12-01T17:04:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e7fff640 1 --2- 172.21.15.67:0/1727812470 >> v2:172.21.15.92:3300/0 conn(0x7fc3e813fe30 0x7fc3e81402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fd636ec9640 10 monclient: shutdown 2023-12-01T17:04:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fd636ec9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:07.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 1 Processor -- start 2023-12-01T17:04:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fd636ec9640 1 -- 172.21.15.67:0/535411133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8e00 msgr2=0x7fd63013f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fd636ec9640 1 --2- 172.21.15.67:0/535411133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8e00 0x7fd63013f800 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd628009d90 tx=0x7fd62802bf50 comp rx=0 tx=0).stop 2023-12-01T17:04:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 1 -- start start 2023-12-01T17:04:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 1 librados: setting wanted keys 2023-12-01T17:04:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 1 librados: calling monclient init 2023-12-01T17:04:07.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7f8f43a61640 10 monclient: init 2023-12-01T17:04:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fd636ec9640 1 -- 172.21.15.67:0/535411133 shutdown_connections 2023-12-01T17:04:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fc3e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fd636ec9640 1 --2- 172.21.15.67:0/535411133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8e00 0x7fd63013f800 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.227+0000 7fd636ec9640 1 -- 172.21.15.67:0/535411133 >> 172.21.15.67:0/535411133 conn(0x7fd6300b7890 msgr2=0x7fd6300b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fd636ec9640 1 -- 172.21.15.67:0/535411133 shutdown_connections 2023-12-01T17:04:07.258 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fd636ec9640 1 -- 172.21.15.67:0/535411133 wait complete. 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0a8a20 0x7f8f3c144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): picked mon.noname-a con 0x7f8f3c0a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c13f0f0 0x7f8f3c0b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): picked mon.noname-b con 0x7f8f3c13f0f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8f3c1461c0 0x7f8f3c0b6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): picked mon.noname-c con 0x7f8f3c1461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c143fd0 con 0x7f8f3c0a8a20 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c1409c0 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3c140c40 con 0x7f8f3c1461c0 2023-12-01T17:04:07.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7f8f43a61640 10 monclient(hunting): authenticate will time out at 1378.466812s 2023-12-01T17:04:07.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 1 -- 172.21.15.67:0/1405442270 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4031870 con 0x7fbcf0145ab0 2023-12-01T17:04:07.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: dump: 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 1 -- 172.21.15.67:0/1405442270 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce40319d0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 1 -- 172.21.15.67:0/1405442270 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce4031d60 con 0x7fbcf0145ab0 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcedffb640 10 monclient: dump: 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.262 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.263 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcf56e5640 5 monclient: authenticate success, global_id 4375 2023-12-01T17:04:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcf56e5640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fbcf56e5640 4 set_mon_vals no callback set 2023-12-01T17:04:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fc3e7fff640 10 monclient(hunting): get_auth_request con 0x7fc3e813fe30 auth_method 0 2023-12-01T17:04:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fc3e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fc3e7fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fc3e7fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fc3e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.228+0000 7fc3e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fc3e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fd636ec9640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fd636ec9640 10 monclient: monmap: 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:07.264 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:07.265 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.265 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.265 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:07.265 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:07.265 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:07.265 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 10 monclient: shutdown 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 -- 172.21.15.67:0/1405442270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf0145ab0 msgr2=0x7fbcf0145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/1405442270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0145ef0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbcf00a9a50 tx=0x7fbce402b910 comp rx=0 tx=0).stop 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 -- 172.21.15.67:0/1405442270 shutdown_connections 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/1405442270 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 0x7fbcf013f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/1405442270 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0145ef0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/1405442270 >> v2:172.21.15.67:3300/0 conn(0x7fbcf00a8a20 0x7fbcf00a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 -- 172.21.15.67:0/1405442270 >> 172.21.15.67:0/1405442270 conn(0x7fbcf00b7890 msgr2=0x7fbcf00b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 -- 172.21.15.67:0/1405442270 shutdown_connections 2023-12-01T17:04:07.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fbcf56e5640 1 -- 172.21.15.67:0/1405442270 wait complete. 2023-12-01T17:04:07.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fd636ec9640 1 librados: starting msgr at 2023-12-01T17:04:07.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fd636ec9640 1 librados: starting objecter 2023-12-01T17:04:07.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.229+0000 7fd636ec9640 1 Processor -- start 2023-12-01T17:04:07.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 -- start start 2023-12-01T17:04:07.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 librados: setting wanted keys 2023-12-01T17:04:07.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 librados: calling monclient init 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient: init 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fbcf56e5640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fbcf56e5640 10 monclient: monmap: 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:07.268 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:07.269 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:07.269 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fbcf56e5640 1 librados: starting msgr at 2023-12-01T17:04:07.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fbcf56e5640 1 librados: starting objecter 2023-12-01T17:04:07.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8a20 0x7fd63013c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): picked mon.noname-a con 0x7fd6300a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6300a8e00 0x7fd63013fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): picked mon.noname-b con 0x7fd6300a8e00 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd6301461c0 0x7fd6301319b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): picked mon.noname-c con 0x7fd6301461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300b6760 con 0x7fd6300a8a20 2023-12-01T17:04:07.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fbcf56e5640 1 Processor -- start 2023-12-01T17:04:07.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300b6e90 con 0x7fd6300a8e00 2023-12-01T17:04:07.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300b6970 con 0x7fd6301461c0 2023-12-01T17:04:07.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.230+0000 7fd636ec9640 10 monclient(hunting): authenticate will time out at 1378.469345s 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fbcf56e5640 1 -- start start 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fbcf56e5640 1 librados: setting wanted keys 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fbcf56e5640 1 librados: calling monclient init 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fbcf56e5640 10 monclient: init 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 10 monclient(hunting): handle_auth_done global_id 4387 payload 274 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 1 -- 172.21.15.67:0/1727812470 >> v2:172.21.15.67:3300/0 conn(0x7fc3e80715f0 msgr2=0x7fc3e813cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 1 --2- 172.21.15.67:0/1727812470 >> v2:172.21.15.67:3300/0 conn(0x7fc3e80715f0 0x7fc3e813cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 1 -- 172.21.15.67:0/1727812470 >> v1:172.21.15.67:6789/0 conn(0x7fc3e80aea80 legacy=0x7fc3e813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:07.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 1 monclient: found mon.noname-c 2023-12-01T17:04:07.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fc3e7fff640 1 -- 172.21.15.67:0/1727812470 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e80a8a80 con 0x7fc3e813fe30 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fbcf56e5640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fbcf56e5640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.231+0000 7fbcf56e5640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf00a8a20 0x7fbcf0143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf00a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 0x7fbcf0135a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf013f0f0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0135fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf0145ab0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:07.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf0136500 con 0x7fbcf0145ab0 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf013cb40 con 0x7fbcf00a8a20 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf013ce20 con 0x7fbcf013f0f0 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.232+0000 7fbcf56e5640 10 monclient(hunting): authenticate will time out at 1378.470809s 2023-12-01T17:04:07.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.233+0000 7fc3e7fff640 1 --2- 172.21.15.67:0/1727812470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e813fe30 0x7fc3e81402a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3e806b570 tx=0x7fc3e0030210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3dcebde928f224b2 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.233+0000 7f8f40ffc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c13f0f0 0x7f8f3c0b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.234+0000 7f8f40ffc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c13f0f0 0x7f8f3c0b5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36804/0 (socket says 172.21.15.67:36804) 2023-12-01T17:04:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.234+0000 7f8f40ffc640 1 -- 172.21.15.67:0/1312846113 learned_addr learned my addr 172.21.15.67:0/1312846113 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.234+0000 7f8f40ffc640 10 monclient(hunting): get_auth_request con 0x7f8f3c13f0f0 auth_method 0 2023-12-01T17:04:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.234+0000 7f8f40ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.234+0000 7f8f40ffc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.234+0000 7f8f40ffc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.235+0000 7fd62ffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6300a8e00 0x7fd63013fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7fd62ffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6300a8e00 0x7fd63013fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36818/0 (socket says 172.21.15.67:36818) 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7fd62ffff640 1 -- 172.21.15.67:0/1888166791 learned_addr learned my addr 172.21.15.67:0/1888166791 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7fd62ffff640 10 monclient(hunting): get_auth_request con 0x7fd6300a8e00 auth_method 0 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7fd62ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7fd62ffff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7fd62ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7f8f417fd640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0a8a20 0x7f8f3c144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.236+0000 7fbceeffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf00a8a20 0x7fbcf0143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fbcef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0135fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd634c65640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8a20 0x7fd63013c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7f8f417fd640 10 monclient(hunting): get_auth_request con 0x7f8f3c0a8a20 auth_method 0 2023-12-01T17:04:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7f8f417fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7f8f417fd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7f8f417fd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd62ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd62ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd62ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fbcef7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0135fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58180/0 (socket says 172.21.15.67:58180) 2023-12-01T17:04:07.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fbcef7fe640 1 -- 172.21.15.67:0/434899083 learned_addr learned my addr 172.21.15.67:0/434899083 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:07.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fbcef7fe640 10 monclient(hunting): get_auth_request con 0x7fbcf0145ab0 auth_method 0 2023-12-01T17:04:07.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fbcef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fbcef7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fbcef7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd634c65640 10 monclient(hunting): get_auth_request con 0x7fd6300a8a20 auth_method 0 2023-12-01T17:04:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd634c65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd634c65640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.237+0000 7fd634c65640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.238+0000 7f8f417fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.238+0000 7f8f417fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.238+0000 7f8f417fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.239+0000 7fbcef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.239+0000 7fbcef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.239+0000 7fbcef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.239+0000 7fd634c65640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.239+0000 7fd634c65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.239+0000 7fd634c65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 10 monclient(hunting): handle_auth_done global_id 4390 payload 824 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 1 -- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8f3c1461c0 msgr2=0x7f8f3c0b6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8f3c1461c0 0x7f8f3c0b6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 1 -- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0a8a20 msgr2=0x7f8f3c144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0a8a20 0x7f8f3c144690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 1 monclient: found mon.noname-b 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.283 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7f8f40ffc640 1 -- 172.21.15.67:0/1312846113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009400 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7fbceeffd640 10 monclient(hunting): get_auth_request con 0x7fbcf00a8a20 auth_method 0 2023-12-01T17:04:07.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7fbceeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7fbceeffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.241+0000 7fbceeffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.242+0000 7fbceeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.242+0000 7fbceeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.242+0000 7fbceeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 1 -- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd6301461c0 msgr2=0x7fd6301319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd6301461c0 0x7fd6301319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 1 -- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8a20 msgr2=0x7fd63013c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8a20 0x7fd63013c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 1 monclient: found mon.noname-b 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.244+0000 7fd62ffff640 1 -- 172.21.15.67:0/1888166791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd628009a70 con 0x7fd6300a8e00 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 1 -- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 msgr2=0x7fbcf0135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 0x7fbcf0135a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 1 -- 172.21.15.67:0/434899083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf00a8a20 msgr2=0x7fbcf0143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf00a8a20 0x7fbcf0143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 1 monclient: found mon.noname-a 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.245+0000 7fbcef7fe640 1 -- 172.21.15.67:0/434899083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce40094d0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.252+0000 7fbcef7fe640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0135fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbce800b970 tx=0x7fbce800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.286 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.252+0000 7fbcd77fe640 10 client.?.objecter ms_handle_connect 0x7fbcf0145ab0 2023-12-01T17:04:07.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.252+0000 7fbcd77fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:07.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.254+0000 7f8f40ffc640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c13f0f0 0x7f8f3c0b5af0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0098c0 tx=0x7f8f2c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.254+0000 7f8f31ffb640 10 client.?.objecter ms_handle_connect 0x7f8f3c13f0f0 2023-12-01T17:04:07.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.254+0000 7f8f31ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:07.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.254+0000 7fd62ffff640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6300a8e00 0x7fd63013fdb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00c400 tx=0x7fd61c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.254+0000 7fd62d7fa640 10 client.?.objecter ms_handle_connect 0x7fd6300a8e00 2023-12-01T17:04:07.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.254+0000 7fd62d7fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:07.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 1 -- 172.21.15.67:0/434899083 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce800cb40 con 0x7fbcf0145ab0 2023-12-01T17:04:07.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:04:07.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: dump: 2023-12-01T17:04:07.289 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.289 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 5 monclient: authenticate success, global_id 4344 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 1 -- 172.21.15.67:0/434899083 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbce800ccc0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 10 monclient: _renew_subs 2023-12-01T17:04:07.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf01366d0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 1 -- 172.21.15.67:0/434899083 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbce8014680 con 0x7fbcf0145ab0 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcd77fe640 10 monclient: dump: 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.291 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbced7fa640 4 set_mon_vals no callback set 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 10 monclient: _renew_subs 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf01443d0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.258+0000 7fbcf56e5640 1 librados: init done 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.259+0000 7fbcf56e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.259+0000 7fbcf56e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.259+0000 7fbcf56e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.259+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf01cb5b0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.292 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.259+0000 7f57977fe640 1 -- 172.21.15.67:0/910698061 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f57900074f0 con 0x7f57a00a8a20 2023-12-01T17:04:07.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 1 -- 172.21.15.67:0/1727812470 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0038440 con 0x7fc3e813fe30 2023-12-01T17:04:07.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:07.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: dump: 2023-12-01T17:04:07.293 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.293 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.293 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 1 -- 172.21.15.67:0/1727812470 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e00385a0 con 0x7fc3e813fe30 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 1 -- 172.21.15.67:0/1727812470 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e0038930 con 0x7fc3e813fe30 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3e67fc640 10 monclient: dump: 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3edffc640 5 monclient: authenticate success, global_id 4387 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3edffc640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:07.295 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3edffc640 4 set_mon_vals no callback set 2023-12-01T17:04:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7f57977fe640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f578006c820 0x7f578006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7f5797fff640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f578006c820 0x7f578006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7f5797fff640 10 monclient: get_auth_request con 0x7f578006c820 auth_method 0 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3edffc640 10 monclient: shutdown 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3edffc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3edffc640 1 -- 172.21.15.67:0/1727812470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e813fe30 msgr2=0x7fc3e81402a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.260+0000 7fc3edffc640 1 --2- 172.21.15.67:0/1727812470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e813fe30 0x7fc3e81402a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc3e806b570 tx=0x7fc3e0030210 comp rx=0 tx=0).stop 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 1 -- 172.21.15.67:0/1727812470 shutdown_connections 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 1 --2- 172.21.15.67:0/1727812470 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e813fe30 0x7fc3e81402a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 1 --2- 172.21.15.67:0/1727812470 >> v2:172.21.15.67:3300/0 conn(0x7fc3e80715f0 0x7fc3e813cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 1 -- 172.21.15.67:0/1727812470 >> 172.21.15.67:0/1727812470 conn(0x7fc3e80b1b60 msgr2=0x7fc3e80b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 1 -- 172.21.15.67:0/1727812470 shutdown_connections 2023-12-01T17:04:07.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 1 -- 172.21.15.67:0/1727812470 wait complete. 2023-12-01T17:04:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7f57977fe640 10 client.4301.objecter ms_handle_connect 0x7f578006c820 2023-12-01T17:04:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7f5797fff640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f578006c820 0x7f578006ece0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57a00b3a00 tx=0x7f5788009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 10 monclient: build_initial_monmap 2023-12-01T17:04:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.261+0000 7fc3edffc640 10 monclient: monmap: 2023-12-01T17:04:07.298 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fc3edffc640 1 librados: starting msgr at 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fc3edffc640 1 librados: starting objecter 2023-12-01T17:04:07.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fc3edffc640 1 Processor -- start 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 1 -- 172.21.15.67:0/1312846113 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c03ba70 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: dump: 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.300 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 1 -- 172.21.15.67:0/1312846113 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c03bbf0 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 1 -- 172.21.15.67:0/1312846113 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0453f0 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:07.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f31ffb640 10 monclient: dump: 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.302 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fc3edffc640 1 -- start start 2023-12-01T17:04:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fc3edffc640 1 librados: setting wanted keys 2023-12-01T17:04:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fc3edffc640 1 librados: calling monclient init 2023-12-01T17:04:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fc3edffc640 10 monclient: init 2023-12-01T17:04:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 5 monclient: authenticate success, global_id 4390 2023-12-01T17:04:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 10 monclient: _renew_subs 2023-12-01T17:04:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f337fe640 4 set_mon_vals no callback set 2023-12-01T17:04:07.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f3c140f50 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 10 monclient: _renew_subs 2023-12-01T17:04:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3c1450d0 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.304 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7f8f43a61640 1 librados: init done 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 1 -- 172.21.15.67:0/1888166791 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c0052d0 con 0x7fd6300a8e00 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: dump: 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.305 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 1 -- 172.21.15.67:0/1888166791 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd61c005450 con 0x7fd6300a8e00 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 5 monclient: authenticate success, global_id 4393 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 10 monclient: _renew_subs 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd630140a00 con 0x7fd6300a8e00 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 1 -- 172.21.15.67:0/1888166791 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd61c003830 con 0x7fd6300a8e00 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:07.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd62d7fa640 10 monclient: dump: 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 10 monclient: _renew_subs 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6301320d0 con 0x7fd6300a8e00 2023-12-01T17:04:07.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.262+0000 7fd636ec9640 1 librados: init done 2023-12-01T17:04:07.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fd62effd640 4 set_mon_vals no callback set 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fc3edffc640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fc3edffc640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fc3edffc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3e80715f0 0x7fc3e80b35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fc3edffc640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e80715f0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fc3edffc640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e80aea80 0x7fc3e81d2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fc3edffc640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e80aea80 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.263+0000 7fc3edffc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e813fe30 0x7fc3e81d3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e813fe30 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e81d3870 con 0x7fc3e813fe30 2023-12-01T17:04:07.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fd636ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fd636ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fd636ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd610005080 con 0x7fd6300a8e00 2023-12-01T17:04:07.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e80a9d90 con 0x7fc3e80aea80 2023-12-01T17:04:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e8143740 con 0x7fc3e80715f0 2023-12-01T17:04:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7f8f43a61640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7f8f43a61640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7f8f43a61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:07.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f3c144140 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.264+0000 7fc3edffc640 10 monclient(hunting): authenticate will time out at 1378.503143s 2023-12-01T17:04:07.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.280+0000 7fc3e7fff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e813fe30 0x7fc3e81d3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.281+0000 7fbcd77fe640 1 -- 172.21.15.67:0/434899083 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbce801d3b0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.281+0000 7fbcd77fe640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbcdc06c820 0x7fbcdc06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.281+0000 7fbceeffd640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbcdc06c820 0x7fbcdc06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.281+0000 7fbceeffd640 10 monclient: get_auth_request con 0x7fbcdc06c820 auth_method 0 2023-12-01T17:04:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.282+0000 7fbceeffd640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbcdc06c820 0x7fbcdc06ece0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbce00098e0 tx=0x7fbce0009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.282+0000 7fbcd77fe640 10 client.4344.objecter ms_handle_connect 0x7fbcdc06c820 2023-12-01T17:04:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 1 -- 172.21.15.67:0/910698061 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5790057dc0 con 0x7f57a00a8a20 2023-12-01T17:04:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 10 client.4301.objecter ms_dispatch 0x7f57a0000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 3 client.4301.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:07.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 3 client.4301.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:07.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 20 client.4301.objecter dump_active .. 0 homeless 2023-12-01T17:04:07.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 1 -- 172.21.15.67:0/910698061 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5790054100 con 0x7f57a00a8a20 2023-12-01T17:04:07.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.489+0000 7f57977fe640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:07.498 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.497+0000 7fc3e77fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3e80715f0 0x7fc3e80b35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.498+0000 7fc3e77fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3e80715f0 0x7fc3e80b35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:42076/0 (socket says 172.21.15.67:42076) 2023-12-01T17:04:07.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.498+0000 7fc3e77fe640 1 -- 172.21.15.67:0/894830569 learned_addr learned my addr 172.21.15.67:0/894830569 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:07.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.498+0000 7fc3e77fe640 10 monclient(hunting): get_auth_request con 0x7fc3e80715f0 auth_method 0 2023-12-01T17:04:07.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.498+0000 7fc3e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:07.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.498+0000 7fc3e77fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:07.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.498+0000 7fc3e77fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.499+0000 7fc3e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.499+0000 7fc3e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:07.500 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.499+0000 7fc3e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2023-12-01T17:04:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 1 -- 172.21.15.67:0/894830569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e80aea80 msgr2=0x7fc3e81d2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e80aea80 0x7fc3e81d2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 1 -- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e813fe30 msgr2=0x7fc3e81d3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e813fe30 0x7fc3e81d3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 10 monclient: _finish_hunting 0 2023-12-01T17:04:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 1 monclient: found mon.noname-c 2023-12-01T17:04:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.502+0000 7fc3e77fe640 1 -- 172.21.15.67:0/894830569 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e0009a70 con 0x7fc3e80715f0 2023-12-01T17:04:07.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.504+0000 7fc3e77fe640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3e80715f0 0x7fc3e80b35a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009fd0 tx=0x7fc3d800ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.504+0000 7fc3cffff640 10 client.?.objecter ms_handle_connect 0x7fc3e80715f0 2023-12-01T17:04:07.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.504+0000 7fc3cffff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 1 -- 172.21.15.67:0/894830569 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800c940 con 0x7fc3e80715f0 2023-12-01T17:04:07.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: dump: 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: _finish_auth 0 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 1 -- 172.21.15.67:0/894830569 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d800caa0 con 0x7fc3e80715f0 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 5 monclient: authenticate success, global_id 4304 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 10 monclient: _renew_subs 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 1 -- 172.21.15.67:0/894830569 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d800ce30 con 0x7fc3e80715f0 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e8143c40 con 0x7fc3e80715f0 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3e5ffb640 4 set_mon_vals no callback set 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3cffff640 10 monclient: dump: 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:07.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 10 monclient: _renew_subs 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e8143490 con 0x7fc3e80715f0 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.508+0000 7fc3edffc640 1 librados: init done 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.509+0000 7fc3edffc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.509+0000 7fc3edffc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.509+0000 7fc3edffc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.509+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e8143900 con 0x7fc3e80715f0 2023-12-01T17:04:07.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.530+0000 7fc3cffff640 1 -- 172.21.15.67:0/894830569 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc3d8013470 con 0x7fc3e80715f0 2023-12-01T17:04:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.531+0000 7fc3cffff640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc3d406c820 0x7fc3d406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.531+0000 7fc3e6ffd640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc3d406c820 0x7fc3d406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.531+0000 7fc3e6ffd640 10 monclient: get_auth_request con 0x7fc3d406c820 auth_method 0 2023-12-01T17:04:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.532+0000 7fc3e6ffd640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc3d406c820 0x7fc3d406ece0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80b3da0 tx=0x7fc3dc0079b0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.532+0000 7fc3cffff640 10 client.4304.objecter ms_handle_connect 0x7fc3d406c820 2023-12-01T17:04:07.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.534+0000 7fc3cffff640 1 -- 172.21.15.67:0/894830569 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc3d8059430 con 0x7fc3e80715f0 2023-12-01T17:04:07.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.534+0000 7fc3cffff640 10 client.4304.objecter ms_dispatch 0x7fc3e8000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.534+0000 7fc3cffff640 3 client.4304.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.534+0000 7fc3cffff640 3 client.4304.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:07.536 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.534+0000 7fc3cffff640 20 client.4304.objecter dump_active .. 0 homeless 2023-12-01T17:04:07.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 1 -- 172.21.15.67:0/434899083 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbce800ce30 con 0x7fbcf0145ab0 2023-12-01T17:04:07.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 10 client.4344.objecter ms_dispatch 0x7fbcf0000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:07.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 3 client.4344.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:07.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 3 client.4344.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:07.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 20 client.4344.objecter dump_active .. 0 homeless 2023-12-01T17:04:07.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 1 -- 172.21.15.67:0/434899083 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbce8054960 con 0x7fbcf0145ab0 2023-12-01T17:04:07.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.539+0000 7fbcd77fe640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:07.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.678+0000 7f57a513e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-01T17:04:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.678+0000 7f57a513e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-01T17:04:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.678+0000 7f57a513e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:07.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.678+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f57a00726a0 con 0x7f57a00a8a20 2023-12-01T17:04:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.723+0000 7fbcf56e5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-01T17:04:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.723+0000 7fbcf56e5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-01T17:04:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.723+0000 7fbcf56e5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.67:3300/0 2023-12-01T17:04:07.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.723+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbcf01cb5b0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.727+0000 7f8f31ffb640 1 -- 172.21.15.67:0/1312846113 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8f2c04e450 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.727+0000 7f8f31ffb640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8f2006c870 0x7f8f2006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.727+0000 7fbcd77fe640 1 -- 172.21.15.67:0/434899083 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbcf01cb5b0 con 0x7fbcf0145ab0 2023-12-01T17:04:07.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.728+0000 7fbcd77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-12-01T17:04:07.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.728+0000 7f8f417fd640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8f2006c870 0x7f8f2006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.728+0000 7f8f417fd640 10 monclient: get_auth_request con 0x7f8f2006c870 auth_method 0 2023-12-01T17:04:07.729 INFO:teuthology.orchestra.run.smithi067.stdout:55834574854 2023-12-01T17:04:07.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.728+0000 7fbcd77fe640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:07.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.728+0000 7f8f417fd640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8f2006c870 0x7f8f2006ed30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f38006fd0 tx=0x7f8f38009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.728+0000 7f8f31ffb640 10 client.4390.objecter ms_handle_connect 0x7f8f2006c870 2023-12-01T17:04:07.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 10 librados: watch_flush enter 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 10 librados: watch_flush exit 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 20 client.4344.objecter shutdown clearing up homeless session... 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 10 client.4344.objecter successfully canceled tick 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbcdc06c820 msgr2=0x7fbcdc06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbcdc06c820 0x7fbcdc06ece0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbce00098e0 tx=0x7fbce0009060 comp rx=0 tx=0).stop 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 10 monclient: shutdown 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf0145ab0 msgr2=0x7fbcf0135fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0135fc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbce800b970 tx=0x7fbce800be30 comp rx=0 tx=0).stop 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 shutdown_connections 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbcf0145ab0 0x7fbcf0135fc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbcdc06c820 0x7fbcdc06ece0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbcf013f0f0 0x7fbcf0135a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 --2- 172.21.15.67:0/434899083 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbcf00a8a20 0x7fbcf0143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 >> 172.21.15.67:0/434899083 conn(0x7fbcf00b7890 msgr2=0x7fbcf00b2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 shutdown_connections 2023-12-01T17:04:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 -- 172.21.15.67:0/434899083 wait complete. 2023-12-01T17:04:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.730+0000 7fbcf56e5640 1 librados: shutdown 2023-12-01T17:04:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.731+0000 7f8f31ffb640 1 -- 172.21.15.67:0/1312846113 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8f2c07b0b0 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.733 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.731+0000 7f8f31ffb640 10 client.4390.objecter ms_dispatch 0x7f8f3c000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:07.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.731+0000 7f8f31ffb640 3 client.4390.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:07.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.731+0000 7f8f31ffb640 3 client.4390.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:07.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.731+0000 7f8f31ffb640 20 client.4390.objecter dump_active .. 0 homeless 2023-12-01T17:04:07.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2023-12-01T17:04:07.743 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:07.754 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.753+0000 7fd62d7fa640 1 -- 172.21.15.67:0/1888166791 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fd61c003990 con 0x7fd6300a8e00 2023-12-01T17:04:07.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.753+0000 7fd62d7fa640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd61406c820 0x7fd61406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:07.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.754+0000 7fd634c65640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd61406c820 0x7fd61406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:07.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.754+0000 7fd634c65640 10 monclient: get_auth_request con 0x7fd61406c820 auth_method 0 2023-12-01T17:04:07.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.754+0000 7fd62d7fa640 10 client.4393.objecter ms_handle_connect 0x7fd61406c820 2023-12-01T17:04:07.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.754+0000 7fd634c65640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd61406c820 0x7fd61406ece0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6240011e0 tx=0x7fd624007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.756+0000 7fd62d7fa640 1 -- 172.21.15.67:0/1888166791 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fd61c057cf0 con 0x7fd6300a8e00 2023-12-01T17:04:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.756+0000 7fd62d7fa640 10 client.4393.objecter ms_dispatch 0x7fd630000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.756+0000 7fd62d7fa640 3 client.4393.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.756+0000 7fd62d7fa640 3 client.4393.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:07.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.756+0000 7fd62d7fa640 20 client.4393.objecter dump_active .. 0 homeless 2023-12-01T17:04:07.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.765+0000 7f8f31ffb640 1 -- 172.21.15.67:0/1312846113 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f2c076cf0 con 0x7f8f3c13f0f0 2023-12-01T17:04:07.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.765+0000 7f8f31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.765+0000 7f8f31ffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:07.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.766+0000 7fd62d7fa640 1 -- 172.21.15.67:0/1888166791 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd61c054030 con 0x7fd6300a8e00 2023-12-01T17:04:07.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.766+0000 7fd62d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.766+0000 7fd62d7fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:07.797 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.796+0000 7fc3cffff640 1 -- 172.21.15.67:0/894830569 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3d8055070 con 0x7fc3e80715f0 2023-12-01T17:04:07.797 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.796+0000 7fc3cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:07.797 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.796+0000 7fc3cffff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:07.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.800+0000 7f57977fe640 1 -- 172.21.15.67:0/910698061 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5790017020 con 0x7f57a00a8a20 2023-12-01T17:04:07.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.800+0000 7f57977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-12-01T17:04:07.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.800+0000 7f57977fe640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:07.801 INFO:teuthology.orchestra.run.smithi067.stdout:60129542150 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 10 librados: watch_flush enter 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 10 librados: watch_flush exit 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 20 client.4301.objecter shutdown clearing up homeless session... 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 10 client.4301.objecter successfully canceled tick 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f578006c820 msgr2=0x7f578006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f578006c820 0x7f578006ece0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57a00b3a00 tx=0x7f5788009040 comp rx=0 tx=0).stop 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 10 monclient: shutdown 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f57a00a8a20 msgr2=0x7f57a0134830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f57a00a8a20 0x7f57a0134830 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f579000c4a0 tx=0x7f579000c960 comp rx=0 tx=0).stop 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 shutdown_connections 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f57a01461c0 0x7f57a013e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f57a00a8e00 0x7f57a0142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f578006c820 0x7f578006ece0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 --2- 172.21.15.67:0/910698061 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f57a00a8a20 0x7f57a0134830 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 >> 172.21.15.67:0/910698061 conn(0x7f57a00b7890 msgr2=0x7f57a00b4260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 shutdown_connections 2023-12-01T17:04:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 -- 172.21.15.67:0/910698061 wait complete. 2023-12-01T17:04:07.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:07.802+0000 7f57a513e640 1 librados: shutdown 2023-12-01T17:04:07.815 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.4 2023-12-01T17:04:07.815 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:08.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.000+0000 7fd636ec9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-01T17:04:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.000+0000 7fd636ec9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-01T17:04:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.000+0000 7fd636ec9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.000+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd610005600 con 0x7fd6300a8e00 2023-12-01T17:04:08.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.003+0000 7fd62d7fa640 1 -- 172.21.15.67:0/1888166791 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd61c017020 con 0x7fd6300a8e00 2023-12-01T17:04:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.003+0000 7fd62d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-12-01T17:04:08.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.003+0000 7fd62d7fa640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:08.005 INFO:teuthology.orchestra.run.smithi067.stdout:73014444039 2023-12-01T17:04:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 10 librados: watch_flush enter 2023-12-01T17:04:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 10 librados: watch_flush exit 2023-12-01T17:04:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 20 client.4393.objecter shutdown clearing up homeless session... 2023-12-01T17:04:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 10 client.4393.objecter successfully canceled tick 2023-12-01T17:04:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd61406c820 msgr2=0x7fd61406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd61406c820 0x7fd61406ece0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd6240011e0 tx=0x7fd624007d30 comp rx=0 tx=0).stop 2023-12-01T17:04:08.007 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 10 monclient: shutdown 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6300a8e00 msgr2=0x7fd63013fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.005+0000 7fd636ec9640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6300a8e00 0x7fd63013fdb0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd61c00c400 tx=0x7fd61c00c8c0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 shutdown_connections 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fd61406c820 0x7fd61406ece0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fd6301461c0 0x7fd6301319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd6300a8e00 0x7fd63013fdb0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 --2- 172.21.15.67:0/1888166791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd6300a8a20 0x7fd63013c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 >> 172.21.15.67:0/1888166791 conn(0x7fd6300b7890 msgr2=0x7fd630144c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 shutdown_connections 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 -- 172.21.15.67:0/1888166791 wait complete. 2023-12-01T17:04:08.008 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.006+0000 7fd636ec9640 1 librados: shutdown 2023-12-01T17:04:08.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.014+0000 7f8f43a61640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-01T17:04:08.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.014+0000 7f8f43a61640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-01T17:04:08.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.014+0000 7f8f43a61640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.014+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f8f3c144140 con 0x7f8f3c13f0f0 2023-12-01T17:04:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.017+0000 7f8f31ffb640 1 -- 172.21.15.67:0/1312846113 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8f3c144140 con 0x7f8f3c13f0f0 2023-12-01T17:04:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.017+0000 7f8f31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-12-01T17:04:08.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.017+0000 7f8f31ffb640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:08.018 INFO:teuthology.orchestra.run.smithi067.stdout:64424509445 2023-12-01T17:04:08.021 INFO:tasks.ceph.ceph_manager.ceph:need seq 73014444039 got 73014444039 for osd.1 2023-12-01T17:04:08.021 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 10 librados: watch_flush enter 2023-12-01T17:04:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 10 librados: watch_flush exit 2023-12-01T17:04:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 20 client.4390.objecter shutdown clearing up homeless session... 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 10 client.4390.objecter successfully canceled tick 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8f2006c870 msgr2=0x7f8f2006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8f2006c870 0x7f8f2006ed30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f8f38006fd0 tx=0x7f8f38009060 comp rx=0 tx=0).stop 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 10 monclient: shutdown 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c13f0f0 msgr2=0x7f8f3c0b5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c13f0f0 0x7f8f3c0b5af0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0098c0 tx=0x7f8f2c0046c0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 shutdown_connections 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8f2006c870 0x7f8f2006ed30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8f3c1461c0 0x7f8f3c0b6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8f3c13f0f0 0x7f8f3c0b5af0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 --2- 172.21.15.67:0/1312846113 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8f3c0a8a20 0x7f8f3c144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 >> 172.21.15.67:0/1312846113 conn(0x7f8f3c0b7890 msgr2=0x7f8f3c13ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 shutdown_connections 2023-12-01T17:04:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 -- 172.21.15.67:0/1312846113 wait complete. 2023-12-01T17:04:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.019+0000 7f8f43a61640 1 librados: shutdown 2023-12-01T17:04:08.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509444 got 64424509445 for osd.0 2023-12-01T17:04:08.032 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.034+0000 7fc3edffc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-01T17:04:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.034+0000 7fc3edffc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-01T17:04:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.034+0000 7fc3edffc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:04:08.035 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.034+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fc3e80a9be0 con 0x7fc3e80715f0 2023-12-01T17:04:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.038+0000 7fc3cffff640 1 -- 172.21.15.67:0/894830569 <== mon.2 v2:172.21.15.67:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc3d8017090 con 0x7fc3e80715f0 2023-12-01T17:04:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.038+0000 7fc3cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-12-01T17:04:08.039 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.038+0000 7fc3cffff640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:08.039 INFO:teuthology.orchestra.run.smithi067.stdout:68719476742 2023-12-01T17:04:08.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 10 librados: watch_flush enter 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 10 librados: watch_flush exit 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 20 client.4304.objecter shutdown clearing up homeless session... 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 10 client.4304.objecter successfully canceled tick 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc3d406c820 msgr2=0x7fc3d406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc3d406c820 0x7fc3d406ece0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3e80b3da0 tx=0x7fc3dc0079b0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 10 monclient: shutdown 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3e80715f0 msgr2=0x7fc3e80b35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3e80715f0 0x7fc3e80b35a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009fd0 tx=0x7fc3d800ed40 comp rx=0 tx=0).stop 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 shutdown_connections 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc3e813fe30 0x7fc3e81d3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc3e80aea80 0x7fc3e81d2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc3d406c820 0x7fc3d406ece0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 --2- 172.21.15.67:0/894830569 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc3e80715f0 0x7fc3e80b35a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 >> 172.21.15.67:0/894830569 conn(0x7fc3e80b1b60 msgr2=0x7fc3e813b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 shutdown_connections 2023-12-01T17:04:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 -- 172.21.15.67:0/894830569 wait complete. 2023-12-01T17:04:08.043 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.040+0000 7fc3edffc640 1 librados: shutdown 2023-12-01T17:04:08.052 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.2 2023-12-01T17:04:08.052 DEBUG:teuthology.parallel:result is None 2023-12-01T17:04:08.052 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-01T17:04:08.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:04:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.198+0000 7fe7f8989640 10 monclient: get_monmap_and_config 2023-12-01T17:04:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.198+0000 7fe7f8989640 10 monclient: build_initial_monmap 2023-12-01T17:04:08.200 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.198+0000 7fe7f8989640 10 monclient: monmap: 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:08.200323+0000 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:08.200323+0000 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:08.201 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 1 Processor -- start 2023-12-01T17:04:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 1 -- start start 2023-12-01T17:04:08.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient: init 2023-12-01T17:04:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-12-01T17:04:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f413eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f40a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7f40a8e00 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe7f41423b0 0x7fe7f4142810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f41423b0 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f40b21f0 con 0x7fe7f40a8e00 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f40a9cc0 con 0x7fe7f41423b0 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f40a9ed0 con 0x7fe7f40a8a20 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.199+0000 7fe7f8989640 10 monclient(hunting): authenticate will time out at 1379.438374s 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f2d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe7f41423b0 0x7fe7f4142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f259c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f413eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7ebfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fe7f40a8e00 0x7fe7f413f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57644/0 (socket says 172.21.15.67:57644) 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7ebfff640 1 -- 172.21.15.67:0/1529238956 learned_addr learned my addr 172.21.15.67:0/1529238956 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f2d9d640 10 monclient(hunting): get_auth_request con 0x7fe7f41423b0 auth_method 0 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f2d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f259c640 10 monclient(hunting): get_auth_request con 0x7fe7f40a8a20 auth_method 0 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f259c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.200+0000 7fe7f259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.201+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.201+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.201+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.201+0000 7fe7f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.201+0000 7fe7f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.207 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.201+0000 7fe7f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.202+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_done global_id 4399 payload 274 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.202+0000 7fe7f2d9d640 1 -- 172.21.15.67:0/1529238956 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 msgr2=0x7fe7f413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.202+0000 7fe7f2d9d640 1 --2- 172.21.15.67:0/1529238956 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f413eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.202+0000 7fe7f2d9d640 1 -- 172.21.15.67:0/1529238956 >> v1:172.21.15.67:6789/0 conn(0x7fe7f40a8e00 legacy=0x7fe7f413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.202+0000 7fe7f2d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.202+0000 7fe7f2d9d640 1 monclient: found mon.noname-c 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.203+0000 7fe7f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.203+0000 7fe7f2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.203+0000 7fe7f2d9d640 1 -- 172.21.15.67:0/1529238956 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f40aac70 con 0x7fe7f41423b0 2023-12-01T17:04:08.208 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.204+0000 7fe7f2d9d640 1 --2- 172.21.15.67:0/1529238956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f4142810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009d90 tx=0x7fe7ec02bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ffcec7b4c2042b9 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 1 -- 172.21.15.67:0/1529238956 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec02cb30 con 0x7fe7f41423b0 2023-12-01T17:04:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: dump: 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: _finish_auth 0 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: _check_auth_tickets 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 1 -- 172.21.15.67:0/1529238956 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec02cc90 con 0x7fe7f41423b0 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f8989640 5 monclient: authenticate success, global_id 4399 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 1 -- 172.21.15.67:0/1529238956 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec034440 con 0x7fe7f41423b0 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f1d9b640 10 monclient: dump: 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.211 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.212 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.212 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.212 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.212 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f8989640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:08.212 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.208+0000 7fe7f8989640 4 set_mon_vals no callback set 2023-12-01T17:04:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 10 monclient: shutdown 2023-12-01T17:04:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:08.215 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 -- 172.21.15.67:0/1529238956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 msgr2=0x7fe7f4142810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 --2- 172.21.15.67:0/1529238956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f4142810 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009d90 tx=0x7fe7ec02bd80 comp rx=0 tx=0).stop 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 -- 172.21.15.67:0/1529238956 shutdown_connections 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 --2- 172.21.15.67:0/1529238956 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f4142810 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 --2- 172.21.15.67:0/1529238956 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f413eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 -- 172.21.15.67:0/1529238956 >> 172.21.15.67:0/1529238956 conn(0x7fe7f40b7870 msgr2=0x7fe7f40b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 -- 172.21.15.67:0/1529238956 shutdown_connections 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.209+0000 7fe7f8989640 1 -- 172.21.15.67:0/1529238956 wait complete. 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient: build_initial_monmap 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient: monmap: 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:08.216 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 librados: starting msgr at 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 librados: starting objecter 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 Processor -- start 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 -- start start 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 librados: setting wanted keys 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 librados: calling monclient init 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient: init 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f41460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f40a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:08.217 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f40a8e00 0x7fe7f413b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f40a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f41daad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f41423b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f413c110 con 0x7fe7f40a8e00 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f413a910 con 0x7fe7f41423b0 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f413ab80 con 0x7fe7f40a8a20 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.210+0000 7fe7f8989640 10 monclient(hunting): authenticate will time out at 1379.449395s 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f259c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f41460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.218 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7ebfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f40a8e00 0x7fe7f413b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f2d9d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f41daad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f259c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f41460f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:42100/0 (socket says 172.21.15.67:42100) 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f259c640 1 -- 172.21.15.67:0/3443527459 learned_addr learned my addr 172.21.15.67:0/3443527459 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f259c640 10 monclient(hunting): get_auth_request con 0x7fe7f40a8a20 auth_method 0 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f259c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f259c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7ebfff640 10 monclient(hunting): get_auth_request con 0x7fe7f40a8e00 auth_method 0 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7ebfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f2d9d640 10 monclient(hunting): get_auth_request con 0x7fe7f41423b0 auth_method 0 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f2d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.219 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.211+0000 7fe7f2d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.220 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.212+0000 7fe7ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2023-12-01T17:04:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 1 -- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 msgr2=0x7fe7f41460f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f41460f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 1 -- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f40a8e00 msgr2=0x7fe7f413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f40a8e00 0x7fe7f413b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:04:08.221 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 1 monclient: found mon.noname-b 2023-12-01T17:04:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7f2d9d640 1 -- 172.21.15.67:0/3443527459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec009a70 con 0x7fe7f41423b0 2023-12-01T17:04:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.215+0000 7fe7ebfff640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f40a8e00 0x7fe7f413b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-01T17:04:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.217+0000 7fe7f2d9d640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f41daad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40b88e0 tx=0x7fe7ec004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.217+0000 7fe7eaffd640 10 client.?.objecter ms_handle_connect 0x7fe7f41423b0 2023-12-01T17:04:08.222 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.217+0000 7fe7eaffd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 1 -- 172.21.15.67:0/3443527459 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec03e410 con 0x7fe7f41423b0 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: dump: 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.223 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: _finish_auth 0 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: _check_auth_tickets 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 1 -- 172.21.15.67:0/3443527459 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec03e5b0 con 0x7fe7f41423b0 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 5 monclient: authenticate success, global_id 4402 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 10 monclient: _renew_subs 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f413ae60 con 0x7fe7f41423b0 2023-12-01T17:04:08.224 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 1 -- 172.21.15.67:0/3443527459 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec03e960 con 0x7fe7f41423b0 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7eaffd640 10 monclient: dump: 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 10 monclient: _renew_subs 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f159a640 4 set_mon_vals no callback set 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f41db010 con 0x7fe7f41423b0 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.221+0000 7fe7f8989640 1 librados: init done 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.222+0000 7fe7f8989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.222+0000 7fe7f8989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.222+0000 7fe7f8989640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.222+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f413bdd0 con 0x7fe7f41423b0 2023-12-01T17:04:08.241 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.240+0000 7fe7eaffd640 1 -- 172.21.15.67:0/3443527459 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe7ec03b050 con 0x7fe7f41423b0 2023-12-01T17:04:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.240+0000 7fe7eaffd640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d406c870 0x7fe7d406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.241+0000 7fe7f259c640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d406c870 0x7fe7d406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.241+0000 7fe7f259c640 10 monclient: get_auth_request con 0x7fe7d406c870 auth_method 0 2023-12-01T17:04:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.241+0000 7fe7f259c640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d406c870 0x7fe7d406ed30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009c00 tx=0x7fe7e4009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:08.242 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.241+0000 7fe7eaffd640 10 client.4402.objecter ms_handle_connect 0x7fe7d406c870 2023-12-01T17:04:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.243+0000 7fe7eaffd640 1 -- 172.21.15.67:0/3443527459 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe7ec07ae00 con 0x7fe7f41423b0 2023-12-01T17:04:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.243+0000 7fe7eaffd640 10 client.4402.objecter ms_dispatch 0x7fe7f4000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.243+0000 7fe7eaffd640 3 client.4402.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.243+0000 7fe7eaffd640 3 client.4402.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:08.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.243+0000 7fe7eaffd640 20 client.4402.objecter dump_active .. 0 homeless 2023-12-01T17:04:08.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.483+0000 7fe7eaffd640 1 -- 172.21.15.67:0/3443527459 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7ec077140 con 0x7fe7f41423b0 2023-12-01T17:04:08.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.483+0000 7fe7eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:08.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.483+0000 7fe7eaffd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:08.664 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.662+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe7f404e840 con 0x7fe7d406c870 2023-12-01T17:04:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.664+0000 7fe7eaffd640 1 -- 172.21.15.67:0/3443527459 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48042 (secure 0 0 0) 0x7fe7f404e840 con 0x7fe7d406c870 2023-12-01T17:04:08.666 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:04:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 10 librados: watch_flush enter 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 10 librados: watch_flush exit 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 20 client.4402.objecter shutdown clearing up homeless session... 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 10 client.4402.objecter successfully canceled tick 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d406c870 msgr2=0x7fe7d406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d406c870 0x7fe7d406ed30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4009c00 tx=0x7fe7e4009190 comp rx=0 tx=0).stop 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 10 monclient: shutdown 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 msgr2=0x7fe7f41daad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f41daad0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe7f40b88e0 tx=0x7fe7ec004a40 comp rx=0 tx=0).stop 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 shutdown_connections 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe7f41423b0 0x7fe7f41daad0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe7f40a8e00 0x7fe7f413b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe7d406c870 0x7fe7d406ed30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 --2- 172.21.15.67:0/3443527459 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe7f40a8a20 0x7fe7f41460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 >> 172.21.15.67:0/3443527459 conn(0x7fe7f40b7870 msgr2=0x7fe7f413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 shutdown_connections 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 -- 172.21.15.67:0/3443527459 wait complete. 2023-12-01T17:04:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.666+0000 7fe7f8989640 1 librados: shutdown 2023-12-01T17:04:08.678 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-12-01T17:04:08.173808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":110,"num_write_kb":1371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218832,"kb_used_data":2480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755888,"statfs":{"total":773094113280,"available":772870029312,"internally_reserved":0,"allocated":2539520,"data_stored":1392888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":11,"num_write_kb":261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.772844+0000","last_change":"2023-12-01T17:04:01.476480+0000","last_active":"2023-12-01T17:04:02.772844+0000","last_peered":"2023-12-01T17:04:02.772844+0000","last_clean":"2023-12-01T17:04:02.772844+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:04:02.772844+0000","last_undegraded":"2023-12-01T17:04:02.772844+0000","last_fullsized":"2023-12-01T17:04:02.772844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:26:16.264231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.738043+0000","last_change":"2023-12-01T17:04:01.536883+0000","last_active":"2023-12-01T17:04:02.738043+0000","last_peered":"2023-12-01T17:04:02.738043+0000","last_clean":"2023-12-01T17:04:02.738043+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:04:02.738043+0000","last_undegraded":"2023-12-01T17:04:02.738043+0000","last_fullsized":"2023-12-01T17:04:02.738043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:42:47.593805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.762034+0000","last_change":"2023-12-01T17:04:01.201853+0000","last_active":"2023-12-01T17:04:02.762034+0000","last_peered":"2023-12-01T17:04:02.762034+0000","last_clean":"2023-12-01T17:04:02.762034+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:04:02.762034+0000","last_undegraded":"2023-12-01T17:04:02.762034+0000","last_fullsized":"2023-12-01T17:04:02.762034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:38:19.219620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:03.125500+0000","last_change":"2023-12-01T17:04:01.467571+0000","last_active":"2023-12-01T17:04:03.125500+0000","last_peered":"2023-12-01T17:04:03.125500+0000","last_clean":"2023-12-01T17:04:03.125500+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:04:03.125500+0000","last_undegraded":"2023-12-01T17:04:03.125500+0000","last_fullsized":"2023-12-01T17:04:03.125500+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:41:15.693504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.758134+0000","last_change":"2023-12-01T17:04:01.438358+0000","last_active":"2023-12-01T17:04:02.758134+0000","last_peered":"2023-12-01T17:04:02.758134+0000","last_clean":"2023-12-01T17:04:02.758134+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:04:02.758134+0000","last_undegraded":"2023-12-01T17:04:02.758134+0000","last_fullsized":"2023-12-01T17:04:02.758134+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:45:24.963538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:03.181039+0000","last_change":"2023-12-01T17:04:01.107520+0000","last_active":"2023-12-01T17:04:03.181039+0000","last_peered":"2023-12-01T17:04:03.181039+0000","last_clean":"2023-12-01T17:04:03.181039+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:04:03.181039+0000","last_undegraded":"2023-12-01T17:04:03.181039+0000","last_fullsized":"2023-12-01T17:04:03.181039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:47:47.456810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.752305+0000","last_change":"2023-12-01T17:04:01.146575+0000","last_active":"2023-12-01T17:04:02.752305+0000","last_peered":"2023-12-01T17:04:02.752305+0000","last_clean":"2023-12-01T17:04:02.752305+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:04:02.752305+0000","last_undegraded":"2023-12-01T17:04:02.752305+0000","last_fullsized":"2023-12-01T17:04:02.752305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:18:50.459254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.764820+0000","last_change":"2023-12-01T17:04:01.495183+0000","last_active":"2023-12-01T17:04:02.764820+0000","last_peered":"2023-12-01T17:04:02.764820+0000","last_clean":"2023-12-01T17:04:02.764820+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:04:02.764820+0000","last_undegraded":"2023-12-01T17:04:02.764820+0000","last_fullsized":"2023-12-01T17:04:02.764820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:13:19.704857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'74","reported_seq":114,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:04.739940+0000","last_change":"2023-12-01T17:03:54.894752+0000","last_active":"2023-12-01T17:04:04.739940+0000","last_peered":"2023-12-01T17:04:04.739940+0000","last_clean":"2023-12-01T17:04:04.739940+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:04:04.739940+0000","last_undegraded":"2023-12-01T17:04:04.739940+0000","last_fullsized":"2023-12-01T17:04:04.739940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_clean_scrub_stamp":"2023-12-01T17:03:51.457677+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:12:02.023301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":110,"num_write_kb":1371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":110,"num_write_kb":1371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":472491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":141.40299999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":248.52500000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.951000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.023}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.435}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":193.768}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.603000000000002}]}]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.056999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.85}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9540000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.148}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.219999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.853}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.183}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.696000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.218000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.391999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.285}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.510999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.646000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.158000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.158999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.882000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.902000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":35.347000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.366}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.628}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.305}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.448}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.763999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.103000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.655999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.509}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.704999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":618496,"data_stored":472491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.785}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8510000000000009}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.553000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.658999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.187999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.521999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.073}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":358.85000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":355.51400000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":359.66300000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":366.34100000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":360.46499999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.401999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":399.07999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.399}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.739000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.489999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.151}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3460000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":50.875999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.786000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:04:08.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:04:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.784+0000 7fbe81f89640 10 monclient: get_monmap_and_config 2023-12-01T17:04:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.784+0000 7fbe81f89640 10 monclient: build_initial_monmap 2023-12-01T17:04:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.784+0000 7fbe81f89640 10 monclient: monmap: 2023-12-01T17:04:08.785 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:08.786071+0000 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:08.786071+0000 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:08.786 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 1 Processor -- start 2023-12-01T17:04:08.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 1 -- start start 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient: init 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbe7c0a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fbe7c0a8e00 0x7fbe7c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient(hunting): picked mon.noname-c con 0x7fbe7c0a8e00 addr v2:172.21.15.92:3300/0 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbe7c1423b0 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7c0b21f0 con 0x7fbe7c1423b0 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7c0a9ed0 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7c0a9cc0 con 0x7fbe7c0a8e00 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.785+0000 7fbe81f89640 10 monclient(hunting): authenticate will time out at 1380.024105s 2023-12-01T17:04:08.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.786+0000 7fbe7bfff640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fbe7c1423b0 0x7fbe7c142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:57650/0 (socket says 172.21.15.67:57650) 2023-12-01T17:04:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.786+0000 7fbe7bfff640 1 -- 172.21.15.67:0/3071536143 learned_addr learned my addr 172.21.15.67:0/3071536143 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.786+0000 7fbe7affd640 1 --2- 172.21.15.67:0/3071536143 >> v2:172.21.15.92:3300/0 conn(0x7fbe7c0a8e00 0x7fbe7c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.787+0000 7fbe7affd640 10 monclient(hunting): get_auth_request con 0x7fbe7c0a8e00 auth_method 0 2023-12-01T17:04:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.787+0000 7fbe7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.787+0000 7fbe7affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.787+0000 7fbe7affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7a7fc640 1 -- 172.21.15.67:0/3071536143 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738689627 0 0) 0x7fbe7c0b21f0 con 0x7fbe7c1423b0 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7a7fc640 10 monclient(hunting): _negotiate 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7a7fc640 10 monclient(hunting): my global_id is 4356 2023-12-01T17:04:08.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.788+0000 7fbe7a7fc640 1 -- 172.21.15.67:0/3071536143 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe64003520 con 0x7fbe7c1423b0 2023-12-01T17:04:08.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2023-12-01T17:04:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 1 -- 172.21.15.67:0/3071536143 >> v1:172.21.15.92:6789/0 conn(0x7fbe7c0a8a20 legacy=0x7fbe7c13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 1 -- 172.21.15.67:0/3071536143 >> v1:172.21.15.67:6789/0 conn(0x7fbe7c1423b0 legacy=0x7fbe7c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 10 monclient: _finish_hunting 0 2023-12-01T17:04:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 1 monclient: found mon.noname-c 2023-12-01T17:04:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.790+0000 7fbe7affd640 1 -- 172.21.15.67:0/3071536143 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c0aac70 con 0x7fbe7c0a8e00 2023-12-01T17:04:08.794 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.793+0000 7fbe7affd640 1 --2- 172.21.15.67:0/3071536143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8e00 0x7fbe7c13f5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe7002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3bbaa3bac5bcc59f server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 1 -- 172.21.15.67:0/3071536143 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7002e070 con 0x7fbe7c0a8e00 2023-12-01T17:04:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:04:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: dump: 2023-12-01T17:04:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.800 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 1 -- 172.21.15.67:0/3071536143 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe700315a0 con 0x7fbe7c0a8e00 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 5 monclient: authenticate success, global_id 4405 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:08.801 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 1 -- 172.21.15.67:0/3071536143 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70031930 con 0x7fbe7c0a8e00 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 4 set_mon_vals no callback set 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe7a7fc640 10 monclient: dump: 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.802 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 10 monclient: shutdown 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 1 -- 172.21.15.67:0/3071536143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8e00 msgr2=0x7fbe7c13f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 1 --2- 172.21.15.67:0/3071536143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8e00 0x7fbe7c13f5c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fbe700097a0 tx=0x7fbe7002b840 comp rx=0 tx=0).stop 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 1 -- 172.21.15.67:0/3071536143 shutdown_connections 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 1 --2- 172.21.15.67:0/3071536143 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8e00 0x7fbe7c13f5c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 1 -- 172.21.15.67:0/3071536143 >> 172.21.15.67:0/3071536143 conn(0x7fbe7c0b7870 msgr2=0x7fbe7c0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 1 -- 172.21.15.67:0/3071536143 shutdown_connections 2023-12-01T17:04:08.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 1 -- 172.21.15.67:0/3071536143 wait complete. 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 10 monclient: build_initial_monmap 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.799+0000 7fbe81f89640 10 monclient: monmap: 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:08.806 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 librados: starting msgr at 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 librados: starting objecter 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 Processor -- start 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 -- start start 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 librados: setting wanted keys 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 librados: calling monclient init 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 10 monclient: init 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:04:08.807 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8a20 0x7fbe7c133be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 10 monclient(hunting): picked mon.noname-b con 0x7fbe7c0a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7c0a8e00 0x7fbe7c143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.800+0000 7fbe81f89640 10 monclient(hunting): picked mon.noname-a con 0x7fbe7c0a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbe7c1423b0 0x7fbe7c1438d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 10 monclient(hunting): picked mon.noname-c con 0x7fbe7c1423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c145630 con 0x7fbe7c0a8e00 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c134fd0 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c1352b0 con 0x7fbe7c1423b0 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe81f89640 10 monclient(hunting): authenticate will time out at 1380.039436s 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8a20 0x7fbe7c133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7bfff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbe7c1423b0 0x7fbe7c1438d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7affd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7c0a8e00 0x7fbe7c143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8a20 0x7fbe7c133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:36856/0 (socket says 172.21.15.67:36856) 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7b7fe640 1 -- 172.21.15.67:0/1499778284 learned_addr learned my addr 172.21.15.67:0/1499778284 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7b7fe640 10 monclient(hunting): get_auth_request con 0x7fbe7c0a8a20 auth_method 0 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7bfff640 10 monclient(hunting): get_auth_request con 0x7fbe7c1423b0 auth_method 0 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7affd640 10 monclient(hunting): get_auth_request con 0x7fbe7c0a8e00 auth_method 0 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:08.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.801+0000 7fbe7affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.802+0000 7fbe7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 10 monclient(hunting): handle_auth_done global_id 4411 payload 824 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 1 -- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbe7c1423b0 msgr2=0x7fbe7c1438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbe7c1423b0 0x7fbe7c1438d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 1 -- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7c0a8e00 msgr2=0x7fbe7c143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7c0a8e00 0x7fbe7c143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 10 monclient: _finish_hunting 0 2023-12-01T17:04:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 1 monclient: found mon.noname-b 2023-12-01T17:04:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.804+0000 7fbe7b7fe640 1 -- 172.21.15.67:0/1499778284 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009400 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.812 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.807+0000 7fbe7b7fe640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8a20 0x7fbe7c133be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00c430 tx=0x7fbe6c00c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.807+0000 7fbe63fff640 10 client.?.objecter ms_handle_connect 0x7fbe7c0a8a20 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.807+0000 7fbe63fff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 1 -- 172.21.15.67:0/1499778284 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c0052d0 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: dump: 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.813 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: _finish_auth 0 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 1 -- 172.21.15.67:0/1499778284 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe6c005450 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 1 -- 172.21.15.67:0/1499778284 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe6c007260 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 5 monclient: authenticate success, global_id 4411 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe63fff640 10 monclient: dump: 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:08.814 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 10 monclient: _renew_subs 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7c134120 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe797fa640 4 set_mon_vals no callback set 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 10 monclient: _renew_subs 2023-12-01T17:04:08.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7c146380 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.810+0000 7fbe81f89640 1 librados: init done 2023-12-01T17:04:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.811+0000 7fbe81f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.811+0000 7fbe81f89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.811+0000 7fbe81f89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:08.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.811+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7c1457b0 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.831+0000 7fbe63fff640 1 -- 172.21.15.67:0/1499778284 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fbe6c0073c0 con 0x7fbe7c0a8a20 2023-12-01T17:04:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.831+0000 7fbe63fff640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbe6806c870 0x7fbe6806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.831+0000 7fbe7affd640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbe6806c870 0x7fbe6806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:08.832 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.831+0000 7fbe7affd640 10 monclient: get_auth_request con 0x7fbe6806c870 auth_method 0 2023-12-01T17:04:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.831+0000 7fbe7affd640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbe6806c870 0x7fbe6806ed30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbe70036000 tx=0x7fbe7003d040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:08.833 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:08.831+0000 7fbe63fff640 10 client.4411.objecter ms_handle_connect 0x7fbe6806c870 2023-12-01T17:04:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.055+0000 7fbe63fff640 1 -- 172.21.15.67:0/1499778284 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fbe6c057960 con 0x7fbe7c0a8a20 2023-12-01T17:04:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.055+0000 7fbe63fff640 10 client.4411.objecter ms_dispatch 0x7fbe7c000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.055+0000 7fbe63fff640 3 client.4411.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.055+0000 7fbe63fff640 3 client.4411.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.055+0000 7fbe63fff640 20 client.4411.objecter dump_active .. 0 homeless 2023-12-01T17:04:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.065+0000 7fbe63fff640 1 -- 172.21.15.67:0/1499778284 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbe6c053ca0 con 0x7fbe7c0a8a20 2023-12-01T17:04:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.065+0000 7fbe63fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:09.066 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.065+0000 7fbe63fff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:09.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.275+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbe7c13fb50 con 0x7fbe6806c870 2023-12-01T17:04:09.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.277+0000 7fbe63fff640 1 -- 172.21.15.67:0/1499778284 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48042 (secure 0 0 0) 0x7fbe70031a80 con 0x7fbe6806c870 2023-12-01T17:04:09.279 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:04:09.279 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 10 librados: watch_flush enter 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 10 librados: watch_flush exit 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 20 client.4411.objecter shutdown clearing up homeless session... 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 10 client.4411.objecter successfully canceled tick 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbe6806c870 msgr2=0x7fbe6806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbe6806c870 0x7fbe6806ed30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbe70036000 tx=0x7fbe7003d040 comp rx=0 tx=0).stop 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 10 monclient: shutdown 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8a20 msgr2=0x7fbe7c133be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8a20 0x7fbe7c133be0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbe6c00c430 tx=0x7fbe6c00c8f0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 shutdown_connections 2023-12-01T17:04:09.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fbe6806c870 0x7fbe6806ed30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fbe7c1423b0 0x7fbe7c1438d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe7c0a8e00 0x7fbe7c143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 --2- 172.21.15.67:0/1499778284 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fbe7c0a8a20 0x7fbe7c133be0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 >> 172.21.15.67:0/1499778284 conn(0x7fbe7c0b7870 msgr2=0x7fbe7c0a9590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 shutdown_connections 2023-12-01T17:04:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 -- 172.21.15.67:0/1499778284 wait complete. 2023-12-01T17:04:09.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.279+0000 7fbe81f89640 1 librados: shutdown 2023-12-01T17:04:09.291 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-12-01T17:04:08.173808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":110,"num_write_kb":1371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218832,"kb_used_data":2480,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755888,"statfs":{"total":773094113280,"available":772870029312,"internally_reserved":0,"allocated":2539520,"data_stored":1392888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":8,"num_read_kb":7,"num_write":11,"num_write_kb":261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001325"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.772844+0000","last_change":"2023-12-01T17:04:01.476480+0000","last_active":"2023-12-01T17:04:02.772844+0000","last_peered":"2023-12-01T17:04:02.772844+0000","last_clean":"2023-12-01T17:04:02.772844+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:04:02.772844+0000","last_undegraded":"2023-12-01T17:04:02.772844+0000","last_fullsized":"2023-12-01T17:04:02.772844+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:26:16.264231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.738043+0000","last_change":"2023-12-01T17:04:01.536883+0000","last_active":"2023-12-01T17:04:02.738043+0000","last_peered":"2023-12-01T17:04:02.738043+0000","last_clean":"2023-12-01T17:04:02.738043+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:04:02.738043+0000","last_undegraded":"2023-12-01T17:04:02.738043+0000","last_fullsized":"2023-12-01T17:04:02.738043+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:42:47.593805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.762034+0000","last_change":"2023-12-01T17:04:01.201853+0000","last_active":"2023-12-01T17:04:02.762034+0000","last_peered":"2023-12-01T17:04:02.762034+0000","last_clean":"2023-12-01T17:04:02.762034+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:04:02.762034+0000","last_undegraded":"2023-12-01T17:04:02.762034+0000","last_fullsized":"2023-12-01T17:04:02.762034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:38:19.219620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:03.125500+0000","last_change":"2023-12-01T17:04:01.467571+0000","last_active":"2023-12-01T17:04:03.125500+0000","last_peered":"2023-12-01T17:04:03.125500+0000","last_clean":"2023-12-01T17:04:03.125500+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:04:03.125500+0000","last_undegraded":"2023-12-01T17:04:03.125500+0000","last_fullsized":"2023-12-01T17:04:03.125500+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:41:15.693504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.758134+0000","last_change":"2023-12-01T17:04:01.438358+0000","last_active":"2023-12-01T17:04:02.758134+0000","last_peered":"2023-12-01T17:04:02.758134+0000","last_clean":"2023-12-01T17:04:02.758134+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:04:02.758134+0000","last_undegraded":"2023-12-01T17:04:02.758134+0000","last_fullsized":"2023-12-01T17:04:02.758134+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:45:24.963538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:03.181039+0000","last_change":"2023-12-01T17:04:01.107520+0000","last_active":"2023-12-01T17:04:03.181039+0000","last_peered":"2023-12-01T17:04:03.181039+0000","last_clean":"2023-12-01T17:04:03.181039+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:04:03.181039+0000","last_undegraded":"2023-12-01T17:04:03.181039+0000","last_fullsized":"2023-12-01T17:04:03.181039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:47:47.456810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.752305+0000","last_change":"2023-12-01T17:04:01.146575+0000","last_active":"2023-12-01T17:04:02.752305+0000","last_peered":"2023-12-01T17:04:02.752305+0000","last_clean":"2023-12-01T17:04:02.752305+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:04:02.752305+0000","last_undegraded":"2023-12-01T17:04:02.752305+0000","last_fullsized":"2023-12-01T17:04:02.752305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:18:50.459254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:02.764820+0000","last_change":"2023-12-01T17:04:01.495183+0000","last_active":"2023-12-01T17:04:02.764820+0000","last_peered":"2023-12-01T17:04:02.764820+0000","last_clean":"2023-12-01T17:04:02.764820+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:04:02.764820+0000","last_undegraded":"2023-12-01T17:04:02.764820+0000","last_fullsized":"2023-12-01T17:04:02.764820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:13:19.704857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"22'74","reported_seq":114,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-12-01T17:04:04.739940+0000","last_change":"2023-12-01T17:03:54.894752+0000","last_active":"2023-12-01T17:04:04.739940+0000","last_peered":"2023-12-01T17:04:04.739940+0000","last_clean":"2023-12-01T17:04:04.739940+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:04:04.739940+0000","last_undegraded":"2023-12-01T17:04:04.739940+0000","last_fullsized":"2023-12-01T17:04:04.739940+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_clean_scrub_stamp":"2023-12-01T17:03:51.457677+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:12:02.023301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":110,"num_write_kb":1371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":80,"num_write":110,"num_write_kb":1371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27624,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":618496,"data_stored":472491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":141.40299999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":248.52500000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.951000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.023}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.435}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":193.768}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.603000000000002}]}]},{"osd":3,"up_from":16,"seq":68719476744,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.056999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.85}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.9540000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.148}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.219999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.853}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.183}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.696000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":44.218000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.391999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.285}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.510999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":41.646000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.158000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509445,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.158999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.882000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":38.902000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":35.347000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.366}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":37.625}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.628}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.305}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.448}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.763999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.103000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.655999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.509}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.704999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27808,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344032,"statfs":{"total":96636764160,"available":96608288768,"internally_reserved":0,"allocated":618496,"data_stored":472491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.785}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.8510000000000009}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.553000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.658999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":24.187999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.521999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":25.073}]}]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":358.85000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":355.51400000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":359.66300000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":366.34100000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":360.46499999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.401999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":399.07999999999998}]}]},{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":217088,"data_stored":74651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.399}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.739000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":26.489999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.151}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3460000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":50.875999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":23.786000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:04:09.292 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-01T17:04:09.292 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-12-01T17:04:09.293 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-12-01T17:04:09.293 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-12-01T17:04:09.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.396+0000 7fa221f1d640 10 monclient: get_monmap_and_config 2023-12-01T17:04:09.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.396+0000 7fa221f1d640 10 monclient: build_initial_monmap 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.396+0000 7fa221f1d640 10 monclient: monmap: 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:04:09.398269+0000 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:04:09.398269+0000 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:04:09.398 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 1 Processor -- start 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 1 -- start start 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient: init 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa21c0b0b80 addr v1:172.21.15.67:6789/0 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa21c0711e0 addr v1:172.21.15.92:6789/0 2023-12-01T17:04:09.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c142870 0x7fa21c14a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa21c142870 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c072590 con 0x7fa21c0b0b80 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c070d90 con 0x7fa21c0711e0 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0b0f60 con 0x7fa21c142870 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.397+0000 7fa221f1d640 10 monclient(hunting): authenticate will time out at 1380.636341s 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.398+0000 7fa21affd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa21c0711e0 0x7fa21c142160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:57886/0 (socket says 172.21.15.67:57886) 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.398+0000 7fa21affd640 1 -- 172.21.15.67:0/2676853679 learned_addr learned my addr 172.21.15.67:0/2676853679 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:04:09.401 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.398+0000 7fa21bfff640 1 --2- 172.21.15.67:0/2676853679 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c142870 0x7fa21c14a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.398+0000 7fa21bfff640 10 monclient(hunting): get_auth_request con 0x7fa21c142870 auth_method 0 2023-12-01T17:04:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.398+0000 7fa21bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.398+0000 7fa21bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.398+0000 7fa21bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.399+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.399+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:09.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.399+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.400+0000 7fa21a7fc640 1 -- 172.21.15.67:0/2676853679 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785220923 0 0) 0x7fa21c070d90 con 0x7fa21c0711e0 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.400+0000 7fa21a7fc640 10 monclient(hunting): _negotiate 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.400+0000 7fa21a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.400+0000 7fa21a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.400+0000 7fa21a7fc640 10 monclient(hunting): my global_id is 4414 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.400+0000 7fa21a7fc640 1 -- 172.21.15.67:0/2676853679 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa204003520 con 0x7fa21c0711e0 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 1 -- 172.21.15.67:0/2676853679 >> v1:172.21.15.92:6789/0 conn(0x7fa21c0711e0 legacy=0x7fa21c142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21a7fc640 1 -- 172.21.15.67:0/2676853679 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664238380 0 0) 0x7fa21c072590 con 0x7fa21c0b0b80 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 1 -- 172.21.15.67:0/2676853679 >> v1:172.21.15.67:6789/0 conn(0x7fa21c0b0b80 legacy=0x7fa21c13bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 10 monclient: _finish_hunting 0 2023-12-01T17:04:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 1 monclient: found mon.noname-b 2023-12-01T17:04:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:04:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21bfff640 1 -- 172.21.15.67:0/2676853679 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c06ff20 con 0x7fa21c142870 2023-12-01T17:04:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.401+0000 7fa21a7fc640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-01T17:04:09.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.403+0000 7fa21bfff640 1 --2- 172.21.15.67:0/2676853679 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c142870 0x7fa21c14a500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa214009dc0 tx=0x7fa21402be90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bcc9ec93e6760ea server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 1 -- 172.21.15.67:0/2676853679 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa21402cc40 con 0x7fa21c142870 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 10 monclient: dump: 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.408 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 10 monclient: _finish_auth 0 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 1 -- 172.21.15.67:0/2676853679 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21402cda0 con 0x7fa21c142870 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.406+0000 7fa21a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa21a7fc640 1 -- 172.21.15.67:0/2676853679 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa214034660 con 0x7fa21c142870 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa21a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa21a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa21a7fc640 10 monclient: dump: 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:09.409 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 5 monclient: authenticate success, global_id 4316 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 10 monclient: get_monmap_and_config success 2023-12-01T17:04:09.410 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 4 set_mon_vals no callback set 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 10 monclient: shutdown 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 1 -- 172.21.15.67:0/2676853679 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c142870 msgr2=0x7fa21c14a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 1 --2- 172.21.15.67:0/2676853679 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c142870 0x7fa21c14a500 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa214009dc0 tx=0x7fa21402be90 comp rx=0 tx=0).stop 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 1 -- 172.21.15.67:0/2676853679 shutdown_connections 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 1 --2- 172.21.15.67:0/2676853679 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c142870 0x7fa21c14a500 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 1 -- 172.21.15.67:0/2676853679 >> 172.21.15.67:0/2676853679 conn(0x7fa21c0b3ef0 msgr2=0x7fa21c0b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 1 -- 172.21.15.67:0/2676853679 shutdown_connections 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.407+0000 7fa221f1d640 1 -- 172.21.15.67:0/2676853679 wait complete. 2023-12-01T17:04:09.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.408+0000 7fa221f1d640 10 monclient: build_initial_monmap 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.408+0000 7fa221f1d640 10 monclient: monmap: 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:04:09.415 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.408+0000 7fa221f1d640 1 librados: starting msgr at 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.408+0000 7fa221f1d640 1 librados: starting objecter 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.408+0000 7fa221f1d640 1 Processor -- start 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 -- start start 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 librados: setting wanted keys 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 librados: calling monclient init 2023-12-01T17:04:09.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient: init 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient: _reopen_session rank -1 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa21c0711e0 0x7fa21c141160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): picked mon.noname-b con 0x7fa21c0711e0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c0b0b80 0x7fa21c0a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): picked mon.noname-c con 0x7fa21c0b0b80 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21c142870 0x7fa21c0a9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): picked mon.noname-a con 0x7fa21c142870 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c14e1e0 con 0x7fa21c142870 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:09.417 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0aaf00 con 0x7fa21c0711e0 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa21c0ab170 con 0x7fa21c0b0b80 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): _renew_subs 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.409+0000 7fa221f1d640 10 monclient(hunting): authenticate will time out at 1380.648014s 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21affd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c0b0b80 0x7fa21c0a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21c142870 0x7fa21c0a9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21b7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa21c0711e0 0x7fa21c141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21affd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c0b0b80 0x7fa21c0a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:42130/0 (socket says 172.21.15.67:42130) 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21affd640 1 -- 172.21.15.67:0/3558363238 learned_addr learned my addr 172.21.15.67:0/3558363238 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21affd640 10 monclient(hunting): get_auth_request con 0x7fa21c0b0b80 auth_method 0 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:09.418 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21b7fe640 10 monclient(hunting): get_auth_request con 0x7fa21c0711e0 auth_method 0 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21bfff640 10 monclient(hunting): get_auth_request con 0x7fa21c142870 auth_method 0 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.410+0000 7fa21bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:04:09.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.411+0000 7fa21bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 10 monclient(hunting): handle_auth_done global_id 4417 payload 824 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 1 -- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c0b0b80 msgr2=0x7fa21c0a97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c0b0b80 0x7fa21c0a97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 1 -- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21c142870 msgr2=0x7fa21c0a9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21c142870 0x7fa21c0a9d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 10 monclient: _finish_hunting 0 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 1 monclient: found mon.noname-b 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.413+0000 7fa21b7fe640 1 -- 172.21.15.67:0/3558363238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa214009a70 con 0x7fa21c0711e0 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.416+0000 7fa21b7fe640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa21c0711e0 0x7fa21c141160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00c400 tx=0x7fa20c00c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.416+0000 7fa203fff640 10 client.?.objecter ms_handle_connect 0x7fa21c0711e0 2023-12-01T17:04:09.421 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.416+0000 7fa203fff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:04:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 1 -- 172.21.15.67:0/3558363238 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0035f0 con 0x7fa21c0711e0 2023-12-01T17:04:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:04:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: dump: 2023-12-01T17:04:09.422 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: _finish_auth 0 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: _check_auth_tickets 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:04:09.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 1 -- 172.21.15.67:0/3558363238 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c003770 con 0x7fa21c0711e0 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 5 monclient: authenticate success, global_id 4417 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 1 -- 172.21.15.67:0/3558363238 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c003b20 con 0x7fa21c0711e0 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa203fff640 10 monclient: dump: 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:04:09.424 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa219ffb640 4 set_mon_vals no callback set 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 10 monclient: _renew_subs 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c0ab450 con 0x7fa21c0711e0 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 10 monclient: _renew_subs 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c14e950 con 0x7fa21c0711e0 2023-12-01T17:04:09.425 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.419+0000 7fa221f1d640 1 librados: init done 2023-12-01T17:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.420+0000 7fa221f1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.420+0000 7fa221f1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.420+0000 7fa221f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:09.426 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.420+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21c14eae0 con 0x7fa21c0711e0 2023-12-01T17:04:09.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.439+0000 7fa203fff640 1 -- 172.21.15.67:0/3558363238 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa20c003c80 con 0x7fa21c0711e0 2023-12-01T17:04:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.439+0000 7fa203fff640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa20806c870 0x7fa20806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:04:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.439+0000 7fa21affd640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa20806c870 0x7fa20806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:04:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.439+0000 7fa21affd640 10 monclient: get_auth_request con 0x7fa20806c870 auth_method 0 2023-12-01T17:04:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.439+0000 7fa203fff640 10 client.4417.objecter ms_handle_connect 0x7fa20806c870 2023-12-01T17:04:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.439+0000 7fa21affd640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa20806c870 0x7fa20806ed30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2100011e0 tx=0x7fa210009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:04:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.443+0000 7fa203fff640 1 -- 172.21.15.67:0/3558363238 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fa20c057e40 con 0x7fa21c0711e0 2023-12-01T17:04:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.443+0000 7fa203fff640 10 client.4417.objecter ms_dispatch 0x7fa21c000b90 osd_map(22..22 src has 1..22) v4 2023-12-01T17:04:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.443+0000 7fa203fff640 3 client.4417.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:04:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.443+0000 7fa203fff640 3 client.4417.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:04:09.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.443+0000 7fa203fff640 20 client.4417.objecter dump_active .. 0 homeless 2023-12-01T17:04:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.673+0000 7fa203fff640 1 -- 172.21.15.67:0/3558363238 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa20c054180 con 0x7fa21c0711e0 2023-12-01T17:04:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.673+0000 7fa203fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:04:09.675 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.673+0000 7fa203fff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:04:09.891 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.890+0000 7fa221f1d640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-12-01T17:04:09.891 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.890+0000 7fa221f1d640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-12-01T17:04:09.891 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.890+0000 7fa221f1d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:04:09.891 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.890+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa21c04e840 con 0x7fa21c0711e0 2023-12-01T17:04:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.896+0000 7fa203fff640 1 -- 172.21.15.67:0/3558363238 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa20c017040 con 0x7fa21c0711e0 2023-12-01T17:04:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.896+0000 7fa203fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-12-01T17:04:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.896+0000 7fa203fff640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:04:09.897 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:04:09.897 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-01T17:04:09.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 10 librados: watch_flush enter 2023-12-01T17:04:09.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 10 librados: watch_flush exit 2023-12-01T17:04:09.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 20 client.4417.objecter shutdown clearing up homeless session... 2023-12-01T17:04:09.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 10 client.4417.objecter successfully canceled tick 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa20806c870 msgr2=0x7fa20806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa20806c870 0x7fa20806ed30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa2100011e0 tx=0x7fa210009040 comp rx=0 tx=0).stop 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 10 monclient: shutdown 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa21c0711e0 msgr2=0x7fa21c141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa21c0711e0 0x7fa21c141160 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa20c00c400 tx=0x7fa20c00c8c0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 shutdown_connections 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa21c142870 0x7fa21c0a9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa20806c870 0x7fa20806ed30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa21c0b0b80 0x7fa21c0a97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.897+0000 7fa221f1d640 1 --2- 172.21.15.67:0/3558363238 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa21c0711e0 0x7fa21c141160 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.898+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 >> 172.21.15.67:0/3558363238 conn(0x7fa21c0b3ef0 msgr2=0x7fa21c13a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:04:09.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.898+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 shutdown_connections 2023-12-01T17:04:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.898+0000 7fa221f1d640 1 -- 172.21.15.67:0/3558363238 wait complete. 2023-12-01T17:04:09.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:04:09.898+0000 7fa221f1d640 1 librados: shutdown 2023-12-01T17:04:09.910 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-12-01T17:04:09.913 INFO:teuthology.run_tasks:Running task workunit... 2023-12-01T17:04:09.927 INFO:tasks.workunit:Pulling workunits from ref d710c09d2ffe0f0858468330fd3d69f1f4506222 2023-12-01T17:04:09.927 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-01T17:04:09.928 DEBUG:teuthology.orchestra.run.smithi067:> stat -- /home/ubuntu/cephtest/mnt.0 2023-12-01T17:04:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T17:04:09.943 INFO:teuthology.orchestra.run.smithi067.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-12-01T17:04:09.943 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-12-01T17:04:09.998 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-12-01T17:04:09.998 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-12-01T17:04:10.054 INFO:tasks.workunit:timeout=6h 2023-12-01T17:04:10.054 INFO:tasks.workunit:cleanup=True 2023-12-01T17:04:10.055 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d710c09d2ffe0f0858468330fd3d69f1f4506222 2023-12-01T17:04:10.110 INFO:tasks.workunit.client.0.smithi067.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-12-01T17:04:21.673 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:35.453 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.733 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:35.513 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.778 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:35.558 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.778 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:35.538 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.781 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:35.562 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.782 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:35.542 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.784 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:35.565 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.818 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:35.578 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:21.838 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:35.619 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.750 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:55.531 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.798 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:55.558 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.799 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:55.559 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.814 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:01:55.574 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.838 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:55.619 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.870 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:01:55.651 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.912 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:55.693 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.915 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:55.696 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:04:41.939 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:01:55.720 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:00.940 INFO:tasks.workunit.client.0.smithi067.stderr:Note: switching to 'd710c09d2ffe0f0858468330fd3d69f1f4506222'. 2023-12-01T17:05:00.940 INFO:tasks.workunit.client.0.smithi067.stderr: 2023-12-01T17:05:00.940 INFO:tasks.workunit.client.0.smithi067.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:state without impacting any branches by switching back to a branch. 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr: 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr: 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr: git switch -c 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr: 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:Or undo this operation with: 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr: 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr: git switch - 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr: 2023-12-01T17:05:00.941 INFO:tasks.workunit.client.0.smithi067.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-01T17:05:00.942 INFO:tasks.workunit.client.0.smithi067.stderr: 2023-12-01T17:05:00.942 INFO:tasks.workunit.client.0.smithi067.stderr:HEAD is now at d710c09d2ff test/os: Add missing include to Fragmentation_simulator.cc 2023-12-01T17:05:00.945 DEBUG:teuthology.orchestra.run.smithi067:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-12-01T17:05:01.011 INFO:tasks.workunit.client.0.smithi067.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-01T17:05:01.013 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-01T17:05:01.013 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-01T17:05:01.683 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-01T17:05:01.745 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-01T17:05:01.787 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-12-01T17:05:01.789 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-01T17:05:01.789 INFO:tasks.workunit.client.0.smithi067.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-01T17:05:01.840 INFO:tasks.workunit.client.0.smithi067.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-12-01T17:05:01.843 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-12-01T17:05:01.843 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-12-01T17:05:01.871 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:02:15.651 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.874 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:02:15.655 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.887 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2023-12-01T17:05:01.888 INFO:tasks.workunit:Running workunit rados/test.sh... 2023-12-01T17:05:01.889 DEBUG:teuthology.orchestra.run.smithi067:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d710c09d2ffe0f0858468330fd3d69f1f4506222 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2023-12-01T17:05:01.889 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:02:15.668 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.920 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:15.680 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.921 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:15.681 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.934 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:02:15.715 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.935 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:15.695 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.935 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:02:15.716 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.948 INFO:tasks.workunit.client.0.smithi067.stderr:+ parallel=1 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' '' = --serial ']' 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ color= 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -t 1 ']' 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ declare -A pids 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_aio 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_aio' 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_aio 2023-12-01T17:05:01.949 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:01.950 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_aio 2023-12-01T17:05:01.950 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92162 2023-12-01T17:05:01.950 INFO:tasks.workunit.client.0.smithi067.stdout:test api_aio on pid 92162 2023-12-01T17:05:01.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_aio on pid 92162' 2023-12-01T17:05:01.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92162 2023-12-01T17:05:01.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:01.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:01.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2023-12-01T17:05:01.951 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_aio_pp 2023-12-01T17:05:01.951 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_aio_pp' 2023-12-01T17:05:01.952 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:02:15.732 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_aio_pp 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:01.954 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.955 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_aio_pp 2023-12-01T17:05:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92169 2023-12-01T17:05:01.956 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_aio_pp on pid 92169' 2023-12-01T17:05:01.956 INFO:tasks.workunit.client.0.smithi067.stdout:test api_aio_pp on pid 92169 2023-12-01T17:05:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92169 2023-12-01T17:05:01.959 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_io 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_io' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_io 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:01.960 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:01.961 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.962 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_io 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92178 2023-12-01T17:05:01.963 INFO:tasks.workunit.client.0.smithi067.stdout:test api_io on pid 92178 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_io on pid 92178' 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92178 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_io_pp 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_io_pp' 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_io_pp 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:01.967 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:01.968 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:01.969 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.970 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_io_pp 2023-12-01T17:05:01.971 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92189 2023-12-01T17:05:01.971 INFO:tasks.workunit.client.0.smithi067.stdout:test api_io_pp on pid 92189 2023-12-01T17:05:01.976 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_io_pp on pid 92189' 2023-12-01T17:05:01.976 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92189 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_asio 2023-12-01T17:05:01.977 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:01.978 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_asio' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:01.979 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.980 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_asio 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:01.981 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.982 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_asio 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92207 2023-12-01T17:05:01.983 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_asio on pid 92207' 2023-12-01T17:05:01.984 INFO:tasks.workunit.client.0.smithi067.stdout:test api_asio on pid 92207 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92207 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:01.993 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:01.994 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:01.995 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.996 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_list 2023-12-01T17:05:01.997 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92162/exe 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_list' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_list 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:01.998 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:01.999 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.000 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.001 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.002 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_aio.log 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_aio 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92169/exe 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_list 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92231 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_list on pid 92231' 2023-12-01T17:05:02.003 INFO:tasks.workunit.client.0.smithi067.stdout:test api_list on pid 92231 2023-12-01T17:05:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92231 2023-12-01T17:05:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.014 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_lock 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_lock' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.015 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_lock 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.016 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.017 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.018 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.019 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.020 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_aio_pp 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92178/exe 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_aio: /' 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.021 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.022 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_io.log 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_aio_pp: /' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.023 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_io 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.024 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_io: /' 2023-12-01T17:05:02.025 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_aio_pp.log 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_lock 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92260 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_lock on pid 92260' 2023-12-01T17:05:02.026 INFO:tasks.workunit.client.0.smithi067.stdout:test api_lock on pid 92260 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92260 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_lock_pp 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.034 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.035 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_lock_pp' 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92189/exe 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.036 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.037 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.038 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_io_pp.log 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_lock_pp 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_io_pp 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_io_pp: /' 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_lock_pp 2023-12-01T17:05:02.039 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.040 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.041 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.041 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92277 2023-12-01T17:05:02.041 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_lock_pp on pid 92277' 2023-12-01T17:05:02.041 INFO:tasks.workunit.client.0.smithi067.stdout:test api_lock_pp on pid 92277 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92277 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.053 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.054 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.055 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_misc 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.056 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.057 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_misc' 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.058 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_misc 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92207/exe 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92231/exe 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.059 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.060 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_asio.log 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_list 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_asio 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.061 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.062 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.063 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.064 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.064 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.064 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.064 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.064 INFO:tasks.workunit.client.0.smithi067.stdout:test api_misc on pid 92305 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_misc 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92305 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_misc on pid 92305' 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92305 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.068 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_asio: /' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_list.log 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_list: /' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_misc_pp 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_misc_pp' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_misc_pp 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.069 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.070 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_misc_pp 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92321 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_misc_pp on pid 92321' 2023-12-01T17:05:02.071 INFO:tasks.workunit.client.0.smithi067.stdout:test api_misc_pp on pid 92321 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92321 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_tier_pp 2023-12-01T17:05:02.082 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_tier_pp' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.083 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.084 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92260/exe 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.085 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_tier_pp 2023-12-01T17:05:02.086 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.087 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_lock.log 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_lock: /' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.088 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_lock 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.089 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_tier_pp 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92363 2023-12-01T17:05:02.090 INFO:tasks.workunit.client.0.smithi067.stdout:test api_tier_pp on pid 92363 2023-12-01T17:05:02.095 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_tier_pp on pid 92363' 2023-12-01T17:05:02.095 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92363 2023-12-01T17:05:02.095 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.095 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_pool 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_pool' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.096 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_pool 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.097 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.098 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_pool 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92384 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.099 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_pool on pid 92384' 2023-12-01T17:05:02.100 INFO:tasks.workunit.client.0.smithi067.stdout:test api_pool on pid 92384 2023-12-01T17:05:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92384 2023-12-01T17:05:02.108 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.109 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.110 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.111 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_snapshots 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_snapshots' 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_snapshots 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92277/exe 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.112 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.113 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.114 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.115 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_lock_pp 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_snapshots 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.116 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92404 2023-12-01T17:05:02.117 INFO:tasks.workunit.client.0.smithi067.stdout:test api_snapshots on pid 92404 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_snapshots on pid 92404' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_lock_pp.log 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92404 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.123 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_lock_pp: /' 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_snapshots_pp 2023-12-01T17:05:02.124 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_snapshots_pp' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92321/exe 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_snapshots_pp 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.125 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.126 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_misc_pp.log 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_misc_pp 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.127 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_misc_pp: /' 2023-12-01T17:05:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_snapshots_pp 2023-12-01T17:05:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92420 2023-12-01T17:05:02.128 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_snapshots_pp on pid 92420' 2023-12-01T17:05:02.128 INFO:tasks.workunit.client.0.smithi067.stdout:test api_snapshots_pp on pid 92420 2023-12-01T17:05:02.134 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92420 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_stat 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_stat' 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.135 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_stat 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.136 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92305/exe 2023-12-01T17:05:02.137 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.138 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_misc 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.139 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_stat 2023-12-01T17:05:02.140 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92444 2023-12-01T17:05:02.140 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_stat on pid 92444' 2023-12-01T17:05:02.140 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.140 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.140 INFO:tasks.workunit.client.0.smithi067.stdout:test api_stat on pid 92444 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92444 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.149 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_misc.log 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_misc: /' 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.150 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.151 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_stat_pp 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_stat_pp' 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_stat_pp 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92363/exe 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.152 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_tier_pp 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_tier_pp.log 2023-12-01T17:05:02.153 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.154 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.155 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.156 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.157 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.157 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.157 INFO:tasks.workunit.client.0.smithi067.stdout:test api_stat_pp on pid 92465 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_stat_pp 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92465 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_stat_pp on pid 92465' 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92465 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_watch_notify 2023-12-01T17:05:02.166 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_watch_notify' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.167 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_tier_pp: /' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.168 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.169 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.170 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.171 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.172 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.173 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.174 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [----------] Global test environment set-up. 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [----------] 12 tests from AsioRados 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.097+0000 7f3f1da37780 10 monclient: get_monmap_and_config 2023-12-01T17:05:02.175 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.097+0000 7f3f1da37780 10 monclient: build_initial_monmap 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.097+0000 7f3f1da37780 10 monclient: monmap: 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: epoch 0 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: last_changed 0.000000 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: created 0.000000 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: election_strategy: 1 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.098+0000 7f3f1da37780 1 Processor -- start 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.098+0000 7f3f1da37780 1 -- start start 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.098+0000 7f3f1da37780 10 monclient: init 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.098+0000 7f3f1da37780 10 monclient: _reopen_session rank -1 2023-12-01T17:05:02.176 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.098+0000 7f3f1da37780 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.098+0000 7f3f1da37780 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 0x55e7c019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.098+0000 7f3f1da37780 10 monclient(hunting): picked mon.noname-a con 0x55e7c0165930 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 0x55e7c01997a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 10 monclient(hunting): picked mon.noname-c con 0x55e7c0167fa0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 10 monclient(hunting): picked mon.noname-b con 0x55e7c0197250 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 10 monclient(hunting): start opening mon connection 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c0111780 con 0x55e7c0165930 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 10 monclient(hunting): start opening mon connection 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c00ab970 con 0x55e7c0197250 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 10 monclient(hunting): start opening mon connection 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7c00ab610 con 0x55e7c0167fa0 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 10 monclient(hunting): _renew_subs 2023-12-01T17:05:02.177 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.099+0000 7f3f1da37780 10 monclient(hunting): authenticate will time out at 1433.336470s 2023-12-01T17:05:02.188 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_watch_notify 2023-12-01T17:05:02.188 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.188 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.188 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.188 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.188 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.189 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.190 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.191 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92444/exe 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.192 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.193 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.194 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_stat.log 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.195 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_stat 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.196 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92404/exe 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_watch_notify 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92549 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92384/exe 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_watch_notify on pid 92549' 2023-12-01T17:05:02.197 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-test api_watch_notify on pid 92549 2023-12-01T17:05:02.202 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92549 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_stat: /' 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.203 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_pool.log 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_watch_notify_pp 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_watch_notify_pp' 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.204 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.205 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_snapshots.log 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_pool: /' 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_watch_notify_pp 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_pool 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_watch_notify_pp 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92571 2023-12-01T17:05:02.206 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_watch_notify_pp on pid 92571' 2023-12-01T17:05:02.207 INFO:tasks.workunit.client.0.smithi067.stdout:test api_watch_notify_pp on pid 92571 2023-12-01T17:05:02.210 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92571 2023-12-01T17:05:02.210 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.210 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2023-12-01T17:05:02.210 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.210 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_cmd 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_snapshots 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_cmd' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_snapshots: /' 2023-12-01T17:05:02.211 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.212 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stdout:01T17:05:02.106+0000 7f3f1b4e4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.109+0000 7f3f1b4e4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02100f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:35550/0 (socket says 172.21.15.67:35550) 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.109+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/4130010696 learned_addr learned my addr 172.21.15.67:0/4130010696 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.109+0000 7f3f1b4e4640 10 monclient(hunting): get_auth_request con 0x55e7c0197250 auth_method 0 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.109+0000 7f3f1b4e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.109+0000 7f3f1b4e4640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:05:02.213 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.109+0000 7f3f1b4e4640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.113+0000 7f3f1b4e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.113+0000 7f3f1b4e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.113+0000 7f3f1b4e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 10 monclient(hunting): handle_auth_done global_id 4435 payload 274 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/4130010696 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 msgr2=0x55e7c01997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/4130010696 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 0x55e7c01997a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/4130010696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 msgr2=0x55e7c019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/4130010696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 0x55e7c019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 10 monclient: _finish_hunting 0 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 1 monclient: found mon.noname-b 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.115+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/4130010696 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f10005d20 con 0x55e7c0197250 2023-12-01T17:05:02.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.118+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/4130010696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02100f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3f1004a8a0 tx=0x7f3f1004c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa89d93f76081dfb server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:05:02.215 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.215 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.215 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.215 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 1 -- 172.21.15.67:0/4130010696 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10052640 con 0x55e7c0197250 2023-12-01T17:05:02.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: dump: 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: epoch 1 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: min_mon_release 18 (reef) 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: election_strategy: 1 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: _finish_auth 0 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: _check_auth_tickets 2023-12-01T17:05:02.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 1 -- 172.21.15.67:0/4130010696 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f100527a0 con 0x55e7c0197250 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 1 -- 172.21.15.67:0/4130010696 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f10052b10 con 0x55e7c0197250 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f194e0640 10 monclient: dump: 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: epoch 1 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: min_mon_release 18 (reef) 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: election_strategy: 1 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:05:02.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f1da37780 5 monclient: authenticate success, global_id 4435 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f1da37780 10 monclient: get_monmap_and_config success 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.132+0000 7f3f1da37780 4 set_mon_vals no callback set 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.133+0000 7f3f1da37780 10 monclient: shutdown 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.133+0000 7f3f1da37780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.133+0000 7f3f1da37780 1 -- 172.21.15.67:0/4130010696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 msgr2=0x55e7c02100f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.133+0000 7f3f1da37780 1 --2- 172.21.15.67:0/4130010696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02100f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3f1004a8a0 tx=0x7f3f1004c000 comp rx=0 tx=0).stop 2023-12-01T17:05:02.218 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 1 -- 172.21.15.67:0/4130010696 shutdown_connections 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 1 --2- 172.21.15.67:0/4130010696 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02100f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 1 --2- 172.21.15.67:0/4130010696 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 0x55e7c01997a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 1 --2- 172.21.15.67:0/4130010696 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 0x55e7c019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 1 -- 172.21.15.67:0/4130010696 >> 172.21.15.67:0/4130010696 conn(0x55e7c0198020 msgr2=0x55e7c01637d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 1 -- 172.21.15.67:0/4130010696 shutdown_connections 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 1 -- 172.21.15.67:0/4130010696 wait complete. 2023-12-01T17:05:02.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 10 monclient: build_initial_monmap 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.134+0000 7f3f1da37780 10 monclient: monmap: 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: epoch 0 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: last_changed 0.000000 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: created 0.000000 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: min_mon_release 0 (unknown) 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: election_strategy: 1 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.135+0000 7f3f1da37780 1 librados: starting msgr at 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.135+0000 7f3f1da37780 1 librados: starting objecter 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.135+0000 7f3f1da37780 1 Processor -- start 2023-12-01T17:05:02.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.135+0000 7f3f1da37780 1 -- start start 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.135+0000 7f3f1da37780 1 librados: setting wanted keys 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.135+0000 7f3f1da37780 1 librados: calling monclient init 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.135+0000 7f3f1da37780 10 monclient: init 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient: _reopen_session rank -1 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 0x55e7c0210260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): picked mon.noname-a con 0x55e7c0165930 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 0x55e7c02c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:02.221 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): picked mon.noname-c con 0x55e7c0167fa0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:05:02.223 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.223 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.223 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.223 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.223 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.223 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.223 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_cmd 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_cmd 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92600 2023-12-01T17:05:02.224 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_cmd on pid 92600' 2023-12-01T17:05:02.225 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02c6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 cryptotest api_cmd on pid 92600 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92600 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.239 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.240 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.241 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_cmd_pp 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_cmd_pp' 2023-12-01T17:05:02.242 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.243 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.244 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92465/exe 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.245 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_stat_pp 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_stat_pp: /' 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.246 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_stat_pp.log 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.247 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.248 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_cmd_pp 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.249 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.250 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92420/exe 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stdout: rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): picked mon.noname-b con 0x55e7c0197250 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): start opening mon connection 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c008b4e0 con 0x55e7c0165930 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): start opening mon connection 2023-12-01T17:05:02.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7c0220f30 con 0x55e7c0197250 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): start opening mon connection 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x55e7c02c84e0 con 0x55e7c0167fa0 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): _renew_subs 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.136+0000 7f3f1da37780 10 monclient(hunting): authenticate will time out at 1433.374138s 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.156+0000 7f3f1ace3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 0x55e7c0210260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.157+0000 7f3f1ace3640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 0x55e7c0210260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36422/0 (socket says 172.21.15.67:36422) 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.157+0000 7f3f1ace3640 1 -- 172.21.15.67:0/1986628037 learned_addr learned my addr 172.21.15.67:0/1986628037 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.157+0000 7f3f1ace3640 10 monclient(hunting): get_auth_request con 0x55e7c0165930 auth_method 0 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.157+0000 7f3f1ace3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.157+0000 7f3f1ace3640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.157+0000 7f3f1ace3640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.159+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.164+0000 7f3f1a4e2640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 0x55e7c02c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.165+0000 7f3f1a4e2640 10 monclient(hunting): get_auth_request con 0x55e7c0167fa0 auth_method 0 2023-12-01T17:05:02.252 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.165+0000 7f3f1a4e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:05:02.253 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.165+0000 7f3f1a4e2640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:05:02.253 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.165+0000 7f3f1a4e2640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:05:02.253 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.166+0000 7f3f1a4e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:05:02.253 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_cmd_pp 2023-12-01T17:05:02.253 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92636 2023-12-01T17:05:02.253 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01Ttest api_cmd_pp on pid 92636 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_cmd_pp on pid 92636' 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92636 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2023-12-01T17:05:02.262 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.263 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.264 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_snapshots_pp: /' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_snapshots_pp 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.265 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.266 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.267 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92549/exe 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_service 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_service' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.268 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_service 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_watch_notify.log 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_watch_notify 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_watch_notify: /' 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_service 2023-12-01T17:05:02.269 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92672 2023-12-01T17:05:02.270 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_service on pid 92672' 2023-12-01T17:05:02.270 INFO:tasks.workunit.client.0.smithi067.stdout:test api_service on pid 92672 2023-12-01T17:05:02.272 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92672 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_service_pp 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_service_pp' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.273 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.274 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_service_pp 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_service_pp 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92694 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_service_pp on pid 92694' 2023-12-01T17:05:02.275 INFO:tasks.workunit.client.0.smithi067.stdout:test api_service_pp on pid 92694 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92694 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.283 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.284 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.285 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_c_write_operations 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_c_write_operations' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_c_write_operations 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.286 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.287 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.288 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.289 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.289 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_c_write_operations 2023-12-01T17:05:02.289 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92717 2023-12-01T17:05:02.289 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_c_write_operations on pid 92717' 2023-12-01T17:05:02.289 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2023-12-01T17:05:02.289 INFO:tasks.workunit.client.0.smithi067.stdout:test api_c_write_operations on pid 92717 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92717 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_c_read_operations 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_c_read_operations' 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.294 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_c_read_operations 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.295 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.296 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_c_read_operations 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92734 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_c_read_operations on pid 92734' 2023-12-01T17:05:02.297 INFO:tasks.workunit.client.0.smithi067.stdout:test api_c_read_operations on pid 92734 2023-12-01T17:05:02.304 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92734 2023-12-01T17:05:02.304 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.304 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s api_cls_remote_reads 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' api_cls_remote_reads' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo api_cls_remote_reads 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.305 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92571/exe 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.306 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.307 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.308 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.309 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.310 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.311 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=api_cls_remote_reads 2023-12-01T17:05:02.311 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.311 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92749 2023-12-01T17:05:02.311 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.311 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test api_cls_remote_reads on pid 92749' 2023-12-01T17:05:02.311 INFO:tasks.workunit.client.0.smithi067.stdout:test api_cls_remote_reads on pid 92749 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92749 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.318 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s list_parallel 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' list_parallel' 2023-12-01T17:05:02.319 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_watch_notify_pp 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo list_parallel 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.320 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.321 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.322 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=list_parallel 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92769 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test list_parallel on pid 92769' 2023-12-01T17:05:02.323 INFO:tasks.workunit.client.0.smithi067.stdout:test list_parallel on pid 92769 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92769 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.333 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.334 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s open_pools_parallel 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' open_pools_parallel' 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.335 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92600/exe 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.336 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.337 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.338 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.339 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.341 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.342 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.343 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_cmd 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo open_pools_parallel 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=open_pools_parallel 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92791 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test open_pools_parallel on pid 92791' 2023-12-01T17:05:02.344 INFO:tasks.workunit.client.0.smithi067.stdout:test open_pools_parallel on pid 92791 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92791 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.362 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_cmd.log 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s delete_pools_parallel 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' delete_pools_parallel' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.363 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.364 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.365 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.366 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.367 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_cmd: /' 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.368 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.369 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.370 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.371 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92636/exe 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.372 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.373 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo delete_pools_parallel 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.374 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_cmd_pp 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.375 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.376 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_cmd_pp: /' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.377 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=delete_pools_parallel 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.378 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92852 2023-12-01T17:05:02.379 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test delete_pools_parallel on pid 92852' 2023-12-01T17:05:02.379 INFO:tasks.workunit.client.0.smithi067.stdout:test delete_pools_parallel on pid 92852 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92852 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92717/exe 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.387 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s cls 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_c_write_operations 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.388 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.389 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.390 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' cls' 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.391 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.392 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92694/exe 2023-12-01T17:05:02.393 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92734/exe 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_c_write_operations: /' 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.394 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout:17:05:02.166+0000 7f3f1a4e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.166+0000 7f3f1a4e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.175+0000 7f3f1b4e4640 10 monclient(hunting): get_auth_request con 0x55e7c0197250 auth_method 0 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.175+0000 7f3f1b4e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.175+0000 7f3f1b4e4640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.175+0000 7f3f1b4e4640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.175+0000 7f3f1ace3640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.175+0000 7f3f1ace3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.175+0000 7f3f1ace3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1b4e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1b4e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1b4e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2023-12-01T17:05:02.395 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 msgr2=0x55e7c02c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x55e7c0197250 0x55e7c02c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 msgr2=0x55e7c0210260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55e7c0165930 0x55e7c0210260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 10 monclient: _finish_hunting 0 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 1 monclient: found mon.noname-c 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.180+0000 7f3f1a4e2640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7c00d1760 con 0x55e7c0167fa0 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.198+0000 7f3f1a4e2640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 0x55e7c02c6350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3f14005fd0 tx=0x7f3f140084f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.198+0000 7f3f03fff640 10 client.?.objecter ms_handle_connect 0x55e7c0167fa0 2023-12-01T17:05:02.396 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.198+0000 7f3f03fff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:05:02.408 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.408 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.408 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.408 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.408 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_c_read_operations 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_service_pp 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.409 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo cls 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.410 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.411 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.412 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.413 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_c_read_operations: /' 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.414 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_service_pp.log 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_service_pp: /' 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.415 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92672/exe 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.416 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.417 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.418 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.419 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.419 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=cls 2023-12-01T17:05:02.419 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92926 2023-12-01T17:05:02.419 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test cls on pid 92926' 2023-12-01T17:05:02.419 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f03fff640 1 test cls on pid 92926 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=92926 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stdout:-- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f14010640 con 0x55e7c0167fa0 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f03fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f03fff640 10 monclient: dump: 2023-12-01T17:05:02.420 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: epoch 1 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: min_mon_release 18 (reef) 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: election_strategy: 1 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f03fff640 10 monclient: _finish_auth 0 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f03fff640 10 monclient: _check_auth_tickets 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 5 monclient: authenticate success, global_id 4325 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 10 monclient: _renew_subs 2023-12-01T17:05:02.421 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55e7c00b7770 con 0x55e7c0167fa0 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 10 monclient: _renew_subs 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7c00aa920 con 0x55e7c0167fa0 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 1 librados: init done 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.259+0000 7f3f1da37780 10 librados: wait_for_osdmap waiting 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.277+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f14005da0 con 0x55e7c0167fa0 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.277+0000 7f3f03fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.277+0000 7f3f194e0640 4 set_mon_vals no callback set 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.288+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f140109a0 con 0x55e7c0167fa0 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.288+0000 7f3f03fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.288+0000 7f3f03fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.288+0000 7f3f03fff640 10 monclient: dump: 2023-12-01T17:05:02.422 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: epoch 1 2023-12-01T17:05:02.423 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:05:02.423 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.423 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:05:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.438 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.439 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_service: /' 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_service.log 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.440 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_service 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92769/exe 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92749/exe 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.441 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.442 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.443 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s cmd 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ list_parallel: /' 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' cmd' 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_api_cls_remote_reads 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_list_parallel 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_list_parallel.log 2023-12-01T17:05:02.444 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.445 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.446 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.447 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92791/exe 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.448 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.449 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo cmd 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ open_pools_parallel: /' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_open_pools_parallel 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.450 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=cmd 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93031 2023-12-01T17:05:02.451 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.452 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test cmd on pid 93031' 2023-12-01T17:05:02.452 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: min_montest cmd on pid 93031 2023-12-01T17:05:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93031 2023-12-01T17:05:02.453 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s handler_error 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' handler_error' 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo handler_error 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=handler_error 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93040 2023-12-01T17:05:02.454 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test handler_error on pid 93040' 2023-12-01T17:05:02.455 INFO:tasks.workunit.client.0.smithi067.stdout:test handler_error on pid 93040 2023-12-01T17:05:02.461 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93040 2023-12-01T17:05:02.461 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.461 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.461 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s io 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' io' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.462 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.463 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.464 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.465 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92926/exe 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo io 2023-12-01T17:05:02.466 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=io 2023-12-01T17:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93058 2023-12-01T17:05:02.467 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test io on pid 93058' 2023-12-01T17:05:02.467 INFO:tasks.workunit.client.0.smithi067.stdout:test io on pid 93058 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93058 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.472 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_cls 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ cls: /' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_cls.log 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.473 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/92852/exe 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s list 2023-12-01T17:05:02.474 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' list' 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo list 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=list 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93094 2023-12-01T17:05:02.475 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test list on pid 93094' 2023-12-01T17:05:02.476 INFO:tasks.workunit.client.0.smithi067.stdout:test list on pid 93094 2023-12-01T17:05:02.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93094 2023-12-01T17:05:02.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.486 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.486 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.486 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.487 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.488 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_rados_delete_pools_parallel 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ delete_pools_parallel: /' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.489 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s misc 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' misc' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.490 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.491 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.492 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.493 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo misc 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.494 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.495 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.496 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.497 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=misc 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93136 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test misc on pid 93136' 2023-12-01T17:05:02.498 INFO:tasks.workunit.client.0.smithi067.stdout:test misc on pid 93136 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93136 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s pool 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' pool' 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2023-12-01T17:05:02.506 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.507 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.508 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.509 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.510 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo pool 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.511 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93040/exe 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=pool 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93173 2023-12-01T17:05:02.512 INFO:tasks.workunit.client.0.smithi067.stdout:test pool on pid 93173 2023-12-01T17:05:02.518 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test pool on pid 93173' 2023-12-01T17:05:02.518 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93173 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s read_operations 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' read_operations' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.519 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo read_operations 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.520 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.521 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93031/exe 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.522 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_handler_error.log 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.523 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_handler_error 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.524 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_cmd 2023-12-01T17:05:02.525 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=read_operations 2023-12-01T17:05:02.526 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93192 2023-12-01T17:05:02.526 INFO:tasks.workunit.client.0.smithi067.stdout:test read_operations on pid 93192 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test read_operations on pid 93192' 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93192 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_cmd.log 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ handler_error: /' 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.546 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.547 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.548 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.549 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.550 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ cmd: /' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s snapshots 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' snapshots' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.551 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.552 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93058/exe 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.553 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.554 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_io 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.555 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_io.log 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ io: /' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.556 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo snapshots 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.557 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.558 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.559 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93094/exe 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=snapshots 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93241 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test snapshots on pid 93241' 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.560 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.561 INFO:tasks.workunit.client.0.smithi067.stdout:test snapshots on pid 93241 2023-12-01T17:05:02.567 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.567 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.567 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.567 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.567 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93241 2023-12-01T17:05:02.567 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s watch_notify 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' watch_notify' 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.568 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo watch_notify 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_list 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.569 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_list.log 2023-12-01T17:05:02.570 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ list: /' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.571 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=watch_notify 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93266 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test watch_notify on pid 93266' 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.572 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.573 INFO:tasks.workunit.client.0.smithi067.stdout:test watch_notify on pid 93266 2023-12-01T17:05:02.580 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93266 2023-12-01T17:05:02.580 INFO:tasks.workunit.client.0.smithi067.stderr:+ for f in cls cmd handler_error io list misc pool read_operations snapshots watch_notify write_operations 2023-12-01T17:05:02.580 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.580 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2023-12-01T17:05:02.580 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.580 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93173/exe 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++ printf %25s write_operations 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+ r=' write_operations' 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.581 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.582 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ awk '{print $1}' 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.583 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93192/exe 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.584 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ pool: /' 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_pool 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:++ echo write_operations 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_pool.log 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.585 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.586 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stderr:+ ff=write_operations 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93289 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stderr:+ echo 'test write_operations on pid 93289' 2023-12-01T17:05:02.587 INFO:tasks.workunit.client.0.smithi067.stdout:test write_operations on pid 93289 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ pids[$f]=93289 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ ret=0 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_read_operations 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' 1 -eq 1 ']' 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92363 2023-12-01T17:05:02.610 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92363 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93136/exe 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_read_operations.log 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ read_operations: /' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.611 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.612 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.613 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_misc 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ misc: /' 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_misc.log 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.614 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -f /etc/bashrc ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:+ . /etc/bashrc 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -z '' ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ BASHRCSOURCED=Y 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ shopt -q login_shell 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:+++ umask 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' 0022 -eq 0 ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ SHELL=/bin/bash 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.615 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/bash_completion.sh 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/cobbler.sh 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ COBBLER_SERVER=172.21.0.11 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorgrep.sh 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.616 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.617 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'grep=grep --color=auto' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'egrep=egrep --color=auto' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'fgrep=fgrep --color=auto' 2023-12-01T17:05:02.618 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorls.sh 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' '!' -t 0 ']' 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:+++ return 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorsysstat.sh 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export S_COLORS=auto 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:+++ S_COLORS=auto 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorxzgrep.sh 2023-12-01T17:05:02.619 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.620 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.621 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/colorzgrep.sh 2023-12-01T17:05:02.622 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93266/exe 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ /usr/libexec/grepconf.sh -c 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.623 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_watch_notify 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ watch_notify: /' 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_watch_notify.log 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.624 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zgrep=zgrep --color=auto' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:+++ alias 'zegrep=zegrep --color=auto' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2023-12-01T17:05:02.625 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/debuginfod.sh 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/gawk.sh 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/lang.sh 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93241/exe 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/locale 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.626 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG_backup=en_US.UTF-8 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /etc/locale.conf ']' 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -x /usr/bin/sed ']' 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.627 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_snapshots 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_snapshots.log 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ snapshots: /' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:++++ export LANG=en_US.UTF-8 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:++++ LANG=en_US.UTF-8 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ LANG=en_US.UTF-8 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ unset LANG_backup config 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n '' ']' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -n en_US.UTF-8 ']' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' dumb = linux ']' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2023-12-01T17:05:02.628 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/less.sh 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++ for i in /etc/profile.d/*.sh 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++ '[' '' ']' 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++ . /etc/profile.d/which2.sh 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+++++ readlink /proc/93289/exe 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++++ basename /usr/bin/bash 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_declare='declare -f' 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+++ which_opt=-f 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export which_declare 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+++ export -f which 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset i 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:++ unset -f pathmunge 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2023-12-01T17:05:02.629 INFO:tasks.workunit.client.0.smithi067.stderr:+ export PATH 2023-12-01T17:05:02.630 INFO:tasks.workunit.client.0.smithi067.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2023-12-01T17:05:02.630 INFO:tasks.workunit.client.0.smithi067.stderr:+ unset rc 2023-12-01T17:05:02.630 INFO:tasks.workunit.client.0.smithi067.stderr:+ ceph_test_neorados_write_operations 2023-12-01T17:05:02.630 INFO:tasks.workunit.client.0.smithi067.stderr:+ tee ceph_test_neorados_write_operations.log 2023-12-01T17:05:02.630 INFO:tasks.workunit.client.0.smithi067.stderr:+ sed 's/^/ write_operations: /' 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout:_release 18 (reef) 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: election_strategy: 1 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.329+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f3f14010b00 con 0x55e7c0167fa0 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.330+0000 7f3f03fff640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3f0806aa70 0x7f3f0806cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.330+0000 7f3f1ace3640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3f0806aa70 0x7f3f0806cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:05:02.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.331+0000 7f3f1ace3640 10 monclient: get_auth_request con 0x7f3f0806aa70 auth_method 0 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.331+0000 7f3f1ace3640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3f0806aa70 0x7f3f0806cf30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0096e0 tx=0x7f3f0c00c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.331+0000 7f3f03fff640 10 client.4325.objecter ms_handle_connect 0x7f3f0806aa70 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.410+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f3f14056ea0 con 0x55e7c0167fa0 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.410+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(22..22 src has 1..22) v4 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.410+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [22,22] > 0 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.410+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding full epoch 22 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.410+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.411+0000 7f3f1da37780 10 librados: wait_for_osdmap done waiting 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.411+0000 7f3f1da37780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.411+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x55e7c010a760 con 0x55e7c0167fa0 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.731+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3f140531e0 con 0x55e7c0167fa0 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.731+0000 7f3f03fff640 10 monclient: handle_get_version_reply finishing 1 version 22 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.731+0000 7f3f18cdf640 10 client.4325.objecter _get_latest_version latest 22, have it 2023-12-01T17:05:02.733 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.732+0000 7f3f1da37780 10 client.4325.objecter create_pool name=ceph_test_rados_api_asio 2023-12-01T17:05:02.734 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.732+0000 7f3f1da37780 10 client.4325.objecter pool_op_submit 1 2023-12-01T17:05:02.734 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.732+0000 7f3f1da37780 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:04.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:02.732+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x55e7c011b810 con 0x55e7c0167fa0 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 7 ==== pool_op_reply(tid 1 (0) Success v23) v1 ==== 43+0+0 (secure 0 0 0) 0x7f3f14056c70 con 0x55e7c0167fa0 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 pool_op_reply(tid 1 (0) Success v23) v1 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 10 client.4325.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v23) v1 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 10 client.4325.objecter have request 1 at 0x55e7c0221180 Op: create 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 20 client.4325.objecter waiting for client to reach epoch 23 before calling back 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 10 monclient: _renew_subs 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=23}) v3 -- 0x7f3f08070780 con 0x55e7c0167fa0 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.131+0000 7f3f03fff640 10 client.4325.objecter done 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 9557+0+0 (secure 0 0 0) 0x7f3f1405f0f0 con 0x55e7c0167fa0 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(23..23 src has 1..23) v4 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [23,23] > 22 2023-12-01T17:05:04.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 23 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 10 librados: set snap read head -> 1 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 10 librados: writefull oid=exist nspace= 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c020ad30 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 23 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 0x55e7c02db380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [writefull 0~5 in=5b] tid 2 osd.2 2023-12-01T17:05:04.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:14.141 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 2 2023-12-01T17:05:14.141 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 15 client.4325.objecter _send_op 2 to 22.1 on osd.2 2023-12-01T17:05:14.141 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:2 22.1 22:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) v8 -- 0x55e7c021c9b0 con 0x55e7c020b6e0 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:04.157+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:05.136+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:05.136+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:05.136+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:05.136+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:06.136+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:06.136+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:06.136+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:07.136+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:07.136+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:07.136+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:07.260+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:14.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:08.137+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:08.137+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:08.137+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:09.137+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:09.137+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:09.137+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:10.137+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:10.137+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:10.137+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:11.137+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:11.137+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:11.137+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:12.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:12.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:12.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:12.261+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:14.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:13.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:14.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:13.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:14.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:13.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:14.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:14.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:17.312 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:14.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:17.312 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:14.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:14.167+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 msgr2=0x55e7c02db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.67:6818/2342127620, fault. 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:14.167+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 0x55e7c02db380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:15.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:15.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:15.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:15.141+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:16.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:16.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:16.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.262+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.262+0000 7f3f1cf47640 2 client.4325.objecter tid 2 on osd.2 is laggy 2023-12-01T17:05:17.313 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.262+0000 7f3f1cf47640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.262+0000 7f3f1cf47640 10 monclient: _renew_subs 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.262+0000 7f3f1cf47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.262+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=24}) v3 -- 0x7f3eec002f90 con 0x55e7c0167fa0 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.262+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- ping magic: 0 v1 -- 0x7f3eec003320 con 0x55e7c020b6e0 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 9 ==== osd_map(24..31 src has 1..31) v4 ==== 19922+0+0 (secure 0 0 0) 0x7f3f14054fe0 con 0x55e7c0167fa0 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(24..31 src has 1..31) v4 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [24,31] > 23 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 24 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 24 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:17.314 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.141 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 25 2023-12-01T17:05:18.141 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 25 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 26 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 26 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 27 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 27 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:18.142 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 28 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 28 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 29 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 29 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 30 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 30 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 31 2023-12-01T17:05:18.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:18.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 31 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:18.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:18.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:18.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:17.312+0000 7f3f03fff640 20 client.4325.objecter 2 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:05:22.126 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: Running main() from gmock_main.cc 2023-12-01T17:05:22.126 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2023-12-01T17:05:22.126 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [----------] Global test environment set-up. 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [----------] 1 test from neocls_handler_error 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (19702 ms) 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [----------] 1 test from neocls_handler_error (19702 ms total) 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [----------] Global test environment tear-down 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [==========] 1 test from 1 test suite ran. (19702 ms total) 2023-12-01T17:05:22.127 INFO:tasks.workunit.client.0.smithi067.stdout: handler_error: [ PASSED ] 1 test. 2023-12-01T17:05:22.284 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:18.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:22.284 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:18.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:22.284 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:18.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:22.284 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:19.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:22.284 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:19.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:19.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:20.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:20.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:20.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:21.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:21.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:21.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.263+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.263+0000 7f3f1cf47640 2 client.4325.objecter tid 2 on osd.2 is laggy 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.263+0000 7f3f1cf47640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.263+0000 7f3f1cf47640 10 monclient: _renew_subs 2023-12-01T17:05:22.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.263+0000 7f3f1cf47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.263+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=32}) v3 -- 0x7f3eec0034e0 con 0x55e7c0167fa0 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.263+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- ping magic: 0 v1 -- 0x7f3eec0036d0 con 0x55e7c020b6e0 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 10 ==== osd_map(32..35 src has 1..35) v4 ==== 4296+0+0 (secure 0 0 0) 0x7f3f14086290 con 0x55e7c0167fa0 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(32..35 src has 1..35) v4 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [32,35] > 31 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 32 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 32 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 33 2023-12-01T17:05:22.286 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:23.092 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:02:36.873 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:23.208 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:02:36.989 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:23.332 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:37.092 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:23.336 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:37.096 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:23.403 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:02:37.184 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:23.458 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:02:37.238 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:23.850 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:02:37.610 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:24.120 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:02:37.901 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:24.174 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:02:37.955 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 33 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 34 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 34 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 35 2023-12-01T17:05:27.264 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 35 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:22.284+0000 7f3f03fff640 20 client.4325.objecter 2 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:23.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:23.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:23.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:24.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:24.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:24.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:24.378+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 msgr2=0x55e7c02db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.67:6818/2342127620, fault. 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:24.378+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 0x55e7c02db380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:25.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:27.265 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:25.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:25.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:25.142+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:26.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:26.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:26.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:27.266 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.264+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023- pool: Running main() from gmock_main.cc 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [==========] Running 6 tests from 1 test suite. 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [----------] Global test environment set-up. 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [----------] 6 tests from NeoRadosPools 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ OK ] NeoRadosPools.PoolList (5151 ms) 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (4483 ms) 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4444 ms) 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2023-12-01T17:05:29.683 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (6094 ms) 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (4236 ms) 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2799 ms) 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [----------] 6 tests from NeoRadosPools (27208 ms total) 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [----------] Global test environment tear-down 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [==========] 6 tests from 1 test suite ran. (27209 ms total) 2023-12-01T17:05:29.684 INFO:tasks.workunit.client.0.smithi067.stdout: pool: [ PASSED ] 6 tests. 2023-12-01T17:05:30.142 INFO:tasks.workunit.client.0.smithi067.stdout:12-01T17:05:27.264+0000 7f3f1cf47640 2 client.4325.objecter tid 2 on osd.2 is laggy 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.264+0000 7f3f1cf47640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.264+0000 7f3f1cf47640 10 monclient: _renew_subs 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.264+0000 7f3f1cf47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.264+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=36}) v3 -- 0x7f3eec003840 con 0x55e7c0167fa0 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.264+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- ping magic: 0 v1 -- 0x7f3eec003a90 con 0x55e7c020b6e0 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 11 ==== osd_map(36..38 src has 1..38) v4 ==== 1933+0+0 (secure 0 0 0) 0x7f3f14004e20 con 0x55e7c0167fa0 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(36..38 src has 1..38) v4 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [36,38] > 35 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 36 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 36 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:30.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 37 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 37 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 38 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 38 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.267+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:27.268+0000 7f3f03fff640 20 client.4325.objecter 2 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:28.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:28.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:28.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:29.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:30.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:29.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:30.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:29.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:30.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:30.141+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:32.269 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:30.141+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:32.269 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:30.141+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:31.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:31.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:31.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.264+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.264+0000 7f3f1cf47640 2 client.4325.objecter tid 2 on osd.2 is laggy 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.264+0000 7f3f1cf47640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.264+0000 7f3f1cf47640 10 monclient: _renew_subs 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.264+0000 7f3f1cf47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.264+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=39}) v3 -- 0x7f3eec003840 con 0x55e7c0167fa0 2023-12-01T17:05:32.270 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.264+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- ping magic: 0 v1 -- 0x7f3eec003c60 con 0x55e7c020b6e0 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 12 ==== osd_map(39..41 src has 1..41) v4 ==== 2282+0+0 (secure 0 0 0) 0x7f3f14086630 con 0x55e7c0167fa0 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(39..41 src has 1..41) v4 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [39,41] > 38 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 39 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 39 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 40 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:32.271 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 40 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:32.273 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:32.273 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 41 2023-12-01T17:05:32.273 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:32.273 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 41 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:32.322 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a5920 api_service: [==========] Running 4 tests from 1 test suite. 2023-12-01T17:05:32.322 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [----------] Global test environment set-up. 2023-12-01T17:05:32.322 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [----------] 4 tests from LibRadosService 2023-12-01T17:05:32.323 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2023-12-01T17:05:32.323 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (6026 ms) 2023-12-01T17:05:32.323 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2023-12-01T17:05:32.323 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ OK ] LibRadosService.RegisterLate (52 ms) 2023-12-01T17:05:32.323 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2023-12-01T17:05:32.323 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: cluster: 2023-12-01T17:05:32.323 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: id: 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:05:32.324 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: health: HEALTH_WARN 2023-12-01T17:05:32.324 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 29 pool(s) do not have an application enabled 2023-12-01T17:05:32.324 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.324 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: services: 2023-12-01T17:05:32.324 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-01T17:05:32.324 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: mgr: y(active, since 119s), standbys: x 2023-12-01T17:05:32.324 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: osd: 8 osds: 8 up (since 76s), 8 in (since 2m) 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: laundry: 2 daemons active (1 hosts) 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: data: 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: pools: 37 pools, 1129 pgs 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: pgs: 75.022% pgs unknown 2023-12-01T17:05:32.325 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 24.181% pgs not active 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 847 unknown 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 273 creating+peering 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 9 active+clean 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: cluster: 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: id: 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: health: HEALTH_WARN 2023-12-01T17:05:32.326 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 17 pool(s) do not have an application enabled 2023-12-01T17:05:32.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: services: 2023-12-01T17:05:32.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2023-12-01T17:05:32.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: mgr: y(active, since 2m), standbys: x 2023-12-01T17:05:32.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: osd: 8 osds: 8 up (since 79s), 8 in (since 2m) 2023-12-01T17:05:32.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2023-12-01T17:05:32.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: laundry: 1 daemon active (1 hosts) 2023-12-01T17:05:32.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: data: 2023-12-01T17:05:32.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: pools: 38 pools, 1137 pgs 2023-12-01T17:05:32.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: objects: 2 objects, 449 KiB 2023-12-01T17:05:32.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2023-12-01T17:05:32.330 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: pgs: 60.510% pgs unknown 2023-12-01T17:05:32.330 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 38.698% pgs not active 2023-12-01T17:05:32.330 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 688 unknown 2023-12-01T17:05:32.330 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 440 creating+peering 2023-12-01T17:05:32.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 9 active+clean 2023-12-01T17:05:32.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ OK ] LibRadosService.StatusFormat (3821 ms) 2023-12-01T17:05:32.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ RUN ] LibRadosService.Status 2023-12-01T17:05:32.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ OK ] LibRadosService.Status (20082 ms) 2023-12-01T17:05:32.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [----------] 4 tests from LibRadosService (29981 ms total) 2023-12-01T17:05:32.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: 2023-12-01T17:05:32.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [----------] Global test environment tear-down 2023-12-01T17:05:32.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [==========] 4 tests from 1 test suite ran. (29981 ms total) 2023-12-01T17:05:32.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_service: [ PASSED ] 4 tests. 2023-12-01T17:05:33.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2023-12-01T17:05:33.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [----------] Global test environment set-up. 2023-12-01T17:05:33.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5324 ms) 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (105 ms) 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20182 ms) 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: attempt 0 of 20 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: failed to deregister: 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: { 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "epoch": 11, 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "modified": "2023-12-01T17:05:28.198409+0000", 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "services": { 2023-12-01T17:05:33.387 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "laundry": { 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "daemons": { 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "summary": "", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "close-test-pid92878": { 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "start_epoch": 11, 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "start_stamp": "2023-12-01T17:05:28.017271+0000", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "gid": 4759, 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "addr": "172.21.15.67:0/3313736745", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "metadata": { 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "arch": "x86_64", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_release": "reef", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-7570-gd710c09d (d710c09d2ffe0f0858468330fd3d69f1f4506222) reef (dev)", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_version_short": "18.0.0-7570-gd710c09d", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-12-01T17:05:33.388 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro": "centos", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro_version": "9", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "foo": "bar", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "hostname": "smithi067", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "mem_total_kb": "32488668", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "os": "Linux", 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "this": "that" 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: }, 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "task_status": {} 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: }, 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "pid92939": { 2023-12-01T17:05:33.389 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "start_epoch": 10, 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "start_stamp": "2023-12-01T17:05:12.325512+0000", 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "gid": 4723, 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "addr": "172.21.15.67:0/2624112986", 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "metadata": { 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "arch": "x86_64", 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_release": "reef", 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-7570-gd710c09d (d710c09d2ffe0f0858468330fd3d69f1f4506222) reef (dev)", 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_version_short": "18.0.0-7570-gd710c09d", 2023-12-01T17:05:33.390 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-12-01T17:05:36.363 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:36.310+0000 12256640 -1 osd.3 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:37.344 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:37.322+0000 12256640 -1 osd.3 42 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:37.609 INFO:tasks.workunit.client.0.smithi067.stdout: 1 2023-12-01T17:05:37.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:37.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:32.268+0000 7f3f03fff640 20 client.4325.objecter 2 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:05:37.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:33.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:37.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:33.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:33.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:34.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:34.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:34.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:34.788+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 msgr2=0x55e7c02db380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.67:6818/2342127620, fault. 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:34.788+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 0x55e7c02db380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:35.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:35.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:35.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:37.610 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:35.142+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:36.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:36.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:36.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.265+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.265+0000 7f3f1cf47640 2 client.4325.objecter tid 2 on osd.2 is laggy 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.265+0000 7f3f1cf47640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.265+0000 7f3f1cf47640 10 monclient: _renew_subs 2023-12-01T17:05:37.611 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.265+0000 7f3f1cf47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:37.612 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.265+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=42}) v3 -- 0x7f3eec003840 con 0x55e7c0167fa0 2023-12-01T17:05:37.612 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.265+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- ping magic: 0 v1 -- 0x7f3eec003e30 con 0x55e7c020b6e0 2023-12-01T17:05:37.612 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 13 ==== osd_map(42..44 src has 1..44) v4 ==== 3061+0+0 (secure 0 0 0) 0x7f3f140535b0 con 0x55e7c0167fa0 2023-12-01T17:05:38.143 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(42..44 src has 1..44) v4 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [42,44] > 41 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 42 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 42 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 43 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 43 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 44 2023-12-01T17:05:38.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 44 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.608+0000 7f3f03fff640 20 client.4325.objecter 2 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.643+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 0x55e7c02db380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.652+0000 7f3f1b4e4640 10 monclient: get_auth_request con 0x55e7c020b6e0 auth_method 0 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.678+0000 7f3f1b4e4640 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 0x55e7c02db380 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.678+0000 7f3f03fff640 10 client.4325.objecter ms_handle_connect 0x55e7c020b6e0 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.761+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 1 ==== osd_map(24..43 src has 1..43) v4 ==== 29967+0+0 (crc 0 0 0) 0x7f3f1006b040 con 0x55e7c020b6e0 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.761+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(24..43 src has 1..43) v4 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.761+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map ignoring epochs [24,43] <= 44 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.761+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:37.761+0000 7f3f03fff640 20 client.4325.objecter 2 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:38.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:38.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:38.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:38.912 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:38.867+0000 11254640 -1 osd.2 43 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9.2e6e800e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:39.134 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:39.128+0000 12256640 -1 osd.3 43 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:40.215 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:40.087+0000 11254640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9.2e6e800e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:40.418 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:40.266+0000 12256640 -1 osd.3 44 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:41.104 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:05:41.085+0000 12256640 -1 osd.6 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4477.0:2 7.b 7.5add198b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:41.176 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:41.166+0000 11254640 -1 osd.2 44 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9.2e6e800e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:42.077 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:05:42.062+0000 12256640 -1 osd.6 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4477.0:2 7.b 7.5add198b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:42.173 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:42.156+0000 10a53640 -1 osd.5 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:42.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:38.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:42.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:39.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:42.327 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:39.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:39.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:40.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:40.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:40.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:41.142+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:41.142+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:41.142+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.266+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.266+0000 7f3f1cf47640 2 client.4325.objecter tid 2 on osd.2 is laggy 2023-12-01T17:05:42.328 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.266+0000 7f3f1cf47640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.266+0000 7f3f1cf47640 10 monclient: _renew_subs 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.266+0000 7f3f1cf47640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.266+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=45}) v3 -- 0x7f3eec003840 con 0x55e7c0167fa0 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.266+0000 7f3f1cf47640 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- ping magic: 0 v1 -- 0x7f3eec003a90 con 0x55e7c020b6e0 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== mon.2 v2:172.21.15.67:3301/0 14 ==== osd_map(45..48 src has 1..48) v4 ==== 4234+0+0 (secure 0 0 0) 0x7f3f14080150 con 0x55e7c0167fa0 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(45..48 src has 1..48) v4 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [45,48] > 44 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 45 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 45 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 46 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 46 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:42.329 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:42.808 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:42.727+0000 11254640 -1 osd.2 45 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:42.852 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:42.586+0000 12256640 -1 osd.3 46 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:43.140 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:05:43.116+0000 12256640 -1 osd.6 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4477.0:2 7.b 7:d198bb5a:test-rados-api-smithi067-92541-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:43.373 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:43.313+0000 10a53640 -1 osd.5 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:43.702 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:43.623+0000 12256640 -1 osd.3 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:43.775 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 3 client.4325.objec api_service_pp: "distro": "centos", 2023-12-01T17:05:43.775 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-01T17:05:43.775 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro_version": "9", 2023-12-01T17:05:43.775 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "foo": "bar", 2023-12-01T17:05:43.775 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "hostname": "smithi067", 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "mem_total_kb": "32488668", 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "os": "Linux", 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "this": "that" 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: }, 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "task_status": {} 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: 2023-12-01T17:05:43.776 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: attempt 1 of 20 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: failed to deregister: 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: { 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "epoch": 13, 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "modified": "2023-12-01T17:05:34.203972+0000", 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "services": { 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "laundry": { 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "daemons": { 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "summary": "", 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "close-test-pid92878": { 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "start_epoch": 13, 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "start_stamp": "2023-12-01T17:05:33.484411+0000", 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "gid": 4774, 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "addr": "172.21.15.67:0/1271707609", 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "metadata": { 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "arch": "x86_64", 2023-12-01T17:05:43.777 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_release": "reef", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_version": "ceph version 18.0.0-7570-gd710c09d (d710c09d2ffe0f0858468330fd3d69f1f4506222) reef (dev)", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "ceph_version_short": "18.0.0-7570-gd710c09d", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro": "centos", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "distro_version": "9", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "foo": "bar", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "hostname": "smithi067", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Tue Nov 21 10:17:52 UTC 2023", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "kernel_version": "5.14.0-388.el9.x86_64", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "mem_swap_kb": "0", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "mem_total_kb": "32488668", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "os": "Linux", 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "this": "that" 2023-12-01T17:05:43.778 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: }, 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: "task_status": {} 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: } 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: attempt 2 of 20 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (15857 ms) 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (41468 ms total) 2023-12-01T17:05:43.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: 2023-12-01T17:05:43.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [----------] Global test environment tear-down 2023-12-01T17:05:43.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (41468 ms total) 2023-12-01T17:05:43.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_service_pp: [ PASSED ] 4 tests. 2023-12-01T17:05:44.103 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:43.987+0000 11254640 -1 osd.2 46 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:44.227 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:05:44.163+0000 12256640 -1 osd.6 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4477.0:2 7.b 7:d198bb5a:test-rados-api-smithi067-92541-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:44.381 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:44.377+0000 10a53640 -1 osd.5 48 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:44.671 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:44.660+0000 12256640 -1 osd.3 47 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:44.931 INFO:tasks.workunit.client.0.smithi067.stdout:ter handle_osd_map decoding incremental epoch 47 2023-12-01T17:05:44.931 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:44.931 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 47 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:44.931 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 48 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 10 client.4325.objecter checking op 2 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 48 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:42.327+0000 7f3f03fff640 20 client.4325.objecter 2 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:43.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:43.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:43.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 2 ==== osd_op_reply(2 exist [writefull 0~5] v43'1 uv1 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:44.932 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(2 exist [writefull 0~5] v43'1 uv1 ondisk = 0) v8 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 2 ondisk uv 1 in 22.1 attempt 0 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 2 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 2 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 2 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 10 librados: Objecter returned from writefull r=0 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c020ad30 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:44.933 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:45.467 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:45.430+0000 10a53640 -1 osd.5 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:45.477 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:05:45.457+0000 12256640 -1 osd.6 48 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4477.0:2 7.b 7:d198bb5a:test-rados-api-smithi067-92541-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:45.570 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:45.509+0000 11254640 -1 osd.2 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:45.754 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:05:45.730+0000 11a55640 -1 osd.4 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4391.0:3 37.14 37.2b3e2654 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e26) 2023-12-01T17:05:45.853 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:45.825+0000 12256640 -1 osd.3 48 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:46.534 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:46.532+0000 11254640 -1 osd.2 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:46.615 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:46.484+0000 10a53640 -1 osd.5 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:46.667 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:05:46.654+0000 12256640 -1 osd.6 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4477.0:2 7.b 7:d198bb5a:test-rados-api-smithi067-92541-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:46.904 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:46.896+0000 12256640 -1 osd.3 49 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:47.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:47.144 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [read 0~256] tid 3 osd.2 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 3 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 15 client.4325.objecter _send_op 3 to 22.1 on osd.2 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:3 22.1 22:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c020db00 con 0x55e7c020b6e0 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02c6dd0 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.a7bbaa1 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:47.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid noexist '@22' '@22' [read 0~256] tid 4 osd.2 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 4 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 15 client.4325.objecter _send_op 4 to 22.1 on osd.2 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:4 22.1 22:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c02c9210 con 0x55e7c020b6e0 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:44.931+0000 7f3f1da37780 5 client.4325.objecter 2 in flight 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:45.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:45.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:45.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:45.144+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:46.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:46.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:46.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:47.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:47.525 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:47.520+0000 11254640 -1 osd.2 47 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:47.667 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:47.591+0000 10a53640 -1 osd.5 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.267+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 3 ondisk uv 1 in 22.1 attempt 0 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 5 2023-12-01T17:05:47.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 3 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 3 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 3 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 5 client.4325.objecter 1 in flight 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 4 ondisk uv 0 in 22.1 attempt 0 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 10 client.4325.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 4 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 4 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:47.843 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 4 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2911 ms) 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.841+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c020ad30 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:47.844 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:47.996 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:05:47.920+0000 12256640 -1 osd.6 51 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:2 7.b 7:d198bb5a:test-rados-api-smithi067-92541-2::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:48.148 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [read 0~256] tid 5 osd.2 2023-12-01T17:05:48.148 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 5 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 15 client.4325.objecter _send_op 5 to 22.1 on osd.2 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:5 22.1 22:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c02c6dd0 con 0x55e7c020b6e0 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02c7240 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.a7bbaa1 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid noexist '@22' '@22' [read 0~256] tid 6 osd.2 2023-12-01T17:05:48.149 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 6 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 15 client.4325.objecter _send_op 6 to 22.1 on osd.2 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:6 22.1 22:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c02c76e0 con 0x55e7c020b6e0 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:47.842+0000 7f3f1da37780 5 client.4325.objecter 2 in flight 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 5 ondisk uv 1 in 22.1 attempt 0 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 5 2023-12-01T17:05:48.150 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 5 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 5 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 5 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 5 client.4325.objecter 1 in flight 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 6 ondisk uv 0 in 22.1 attempt 0 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 10 client.4325.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 6 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 6 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 6 2023-12-01T17:05:48.151 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (307 ms) 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02c7240 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [read 0~256] tid 7 osd.2 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 7 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 15 client.4325.objecter _send_op 7 to 22.1 on osd.2 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:7 22.1 22:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c020ad30 con 0x55e7c020b6e0 2023-12-01T17:05:48.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c031bf50 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.a7bbaa1 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid noexist '@22' '@22' [read 0~256] tid 8 osd.2 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 8 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 15 client.4325.objecter _send_op 8 to 22.1 on osd.2 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:8 22.1 22:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c031c3f0 con 0x55e7c020b6e0 2023-12-01T17:05:48.242 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.148+0000 7f3f1da37780 5 client.4325.objecter 2 in flight 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 7 ondisk uv 1 in 22.1 attempt 0 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 5 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 7 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 7 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 7 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.202+0000 7f3f1b4e4640 5 client.4325.objecter 1 in flight 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-01T17:05:48.243 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:48.244 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 8 ondisk uv 0 in 22.1 attempt 0 2023-12-01T17:05:48.261 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:48.185+0000 12256640 -1 osd.3 50 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:48.773 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:02.554 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:48.777 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:02.558 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:48.845 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:02.605 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:48.847 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:02.607 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:48.851 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:48.800+0000 10a53640 -1 osd.5 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:48.857 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:02.638 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:48.860 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:02.641 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:48.888 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:48.827+0000 11254640 -1 osd.2 49 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:49.039 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:02.799 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:49.081 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:02.861 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:49.093 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:02.874 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:05:49.881 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:49.760+0000 12256640 -1 osd.3 51 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:50.136 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:50.121+0000 11254640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:50.376 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:05:50.274+0000 10a53640 -1 osd.5 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4331.0:2 15.6 15:602f83fe:::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25) 2023-12-01T17:05:51.042 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:50.992+0000 12256640 -1 osd.3 52 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 10 client.4325.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 8 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (93 ms) 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 8 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 8 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:05:51.145 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 librados: queue_aio_write 0x55e7c010a480 completion 0x55e7c00a5ca0 write_seq 1 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02c7240 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [write 0~5 in=5b] tid 9 osd.2 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 9 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 15 client.4325.objecter _send_op 9 to 22.1 on osd.2 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:9 22.1 22:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c02dc8d0 con 0x55e7c020b6e0 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:51.146 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:48.241+0000 7f3f1da37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:49.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:49.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:49.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:50.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:50.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:50.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.144+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:51.147 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.144+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:51.179 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:51.170+0000 11254640 -1 osd.2 50 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:51.502 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.144+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 9 ==== osd_op_reply(9 exist [write 0~5] v48'2 uv2 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(9 exist [write 0~5] v48'2 uv2 ondisk = 0) v8 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 9 ondisk uv 2 in 22.1 attempt 0 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 9 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 9 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 9 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1b4e4640 20 librados: complete_aio_write 0x55e7c00a5ca0 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (3158 ms) 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-01T17:05:51.503 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 librados: queue_aio_write 0x55e7c010a480 completion 0x55e7c00b2660 write_seq 2 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02c7240 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 48 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [write 0~5 in=5b] tid 10 osd.2 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 10 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 15 client.4325.objecter _send_op 10 to 22.1 on osd.2 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:10 22.1 22:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e48) v8 -- 0x55e7c02dcc30 con 0x55e7c020b6e0 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:51.504 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.399+0000 7f3f1da37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-01T17:05:52.149 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:52.092+0000 12256640 -1 osd.3 53 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:52.514 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:52.497+0000 11254640 -1 osd.2 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:53.308 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:53.214+0000 12256640 -1 osd.3 54 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:53.615 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:05:53.594+0000 11254640 -1 osd.2 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:2 9.e 9:70017674:test-rados-api-smithi067-92349-2::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:54.493 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:54.470+0000 12256640 -1 osd.3 55 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:55.593 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:55.507+0000 12256640 -1 osd.3 56 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 10 ==== osd_map(49..50 src has 1..54) v4 ==== 951+0+0 (crc 0 0 0) 0x7f3f10066450 con 0x55e7c020b6e0 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(49..50 src has 1..54) v4 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [49,50] > 48 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 49 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 10 client.4325.objecter checking op 10 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 49 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 50 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 10 client.4325.objecter checking op 10 2023-12-01T17:05:55.850 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 50 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:51.502+0000 7f3f03fff640 20 client.4325.objecter 10 22.d3a59201 osd.2 exist [write 0~5 in=5b] 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:52.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:52.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:52.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:52.268+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:53.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:53.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:53.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:54.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:54.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:54.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:55.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:55.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:55.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:55.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.145+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:05:55.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 11 ==== osd_op_reply(10 exist [write 0~5] v50'3 uv3 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:55.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(10 exist [write 0~5] v50'3 uv3 ondisk = 0) v8 2023-12-01T17:05:55.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:55.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 10 ondisk uv 3 in 22.1 attempt 0 2023-12-01T17:05:55.943 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:05:55.943 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 10 2023-12-01T17:05:55.943 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 10 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 10 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1b4e4640 20 librados: complete_aio_write 0x55e7c00b2660 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (4450 ms) 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 librados: queue_aio_write 0x55e7c010a480 completion 0x55e7c02c7240 write_seq 3 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02ff6e0 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 50 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:55.944 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [write 0~5 in=5b] tid 11 osd.2 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 11 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 15 client.4325.objecter _send_op 11 to 22.1 on osd.2 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:11 22.1 22:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e50) v8 -- 0x55e7c02ffb80 con 0x55e7c020b6e0 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.850+0000 7f3f1da37780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 12 ==== osd_map(51..55 src has 1..58) v4 ==== 3966+0+0 (crc 0 0 0) 0x7f3f10052ab0 con 0x55e7c020b6e0 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(51..55 src has 1..58) v4 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [51,55] > 50 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 51 2023-12-01T17:05:55.945 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 10 client.4325.objecter checking op 11 2023-12-01T17:05:55.946 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 51 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:56.836 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:56.831+0000 12256640 -1 osd.3 57 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:57.986 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:05:57.981+0000 11a55640 -1 osd.4 59 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4502.0:2 55.0 55:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-01T17:05:58.069 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:57.997+0000 12256640 -1 osd.3 58 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:59.097 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:05:59.090+0000 11a55640 -1 osd.4 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4502.0:2 55.0 55:0384ab8e:::oid:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e37) 2023-12-01T17:05:59.207 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:05:59.193+0000 12256640 -1 osd.3 59 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 52 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 10 client.4325.objecter checking op 11 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 52 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 53 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 10 client.4325.objecter checking op 11 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 53 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 54 2023-12-01T17:05:59.838 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 10 client.4325.objecter checking op 11 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 54 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 55 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 10 client.4325.objecter checking op 11 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 55 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:55.943+0000 7f3f03fff640 20 client.4325.objecter 11 22.d3a59201 osd.2 exist [write 0~5 in=5b] 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:56.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:56.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:56.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:57.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:57.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:57.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:59.839 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:57.269+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:58.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:58.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:58.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 13 ==== osd_op_reply(11 exist [write 0~5] v55'4 uv4 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(11 exist [write 0~5] v55'4 uv4 ondisk = 0) v8 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 11 ondisk uv 4 in 22.1 attempt 0 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 11 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 11 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:59.840 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 11 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1b4e4640 20 librados: complete_aio_write 0x55e7c02c7240 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3988 ms) 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02db8c0 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 55 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [read 0~0] tid 12 osd.2 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 12 2023-12-01T17:05:59.841 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 15 client.4325.objecter _send_op 12 to 22.1 on osd.2 2023-12-01T17:05:59.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:12 22.1 22:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x55e7c02dbd60 con 0x55e7c020b6e0 2023-12-01T17:05:59.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:05:59.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:05:59.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02dc0c0 2023-12-01T17:05:59.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 55 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:05:59.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:05:59.842 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.a7bbaa1 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:00.152 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid noexist '@22' '@22' [read 0~0] tid 13 osd.2 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 13 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 15 client.4325.objecter _send_op 13 to 22.1 on osd.2 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:13 22.1 22:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x55e7c02dc560 con 0x55e7c020b6e0 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:05:59.837+0000 7f3f1da37780 5 client.4325.objecter 2 in flight 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 14 ==== osd_map(56..59 src has 1..61) v4 ==== 2561+0+0 (crc 0 0 0) 0x7f3f10052ab0 con 0x55e7c020b6e0 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(56..59 src has 1..61) v4 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [56,59] > 55 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 56 2023-12-01T17:06:00.153 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 12 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 56 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 13 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 56 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 57 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 12 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 57 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 13 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 57 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:06:00.154 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 58 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 12 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 58 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 13 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 58 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 59 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 12 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 59 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 10 client.4325.objecter checking op 13 2023-12-01T17:06:00.164 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 59 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter 12 22.d3a59201 osd.2 exist [read 0~0] 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.152+0000 7f3f03fff640 20 client.4325.objecter 13 22.a7bbaa1 osd.2 noexist [read 0~0] 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 15 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 12 ondisk uv 4 in 22.1 attempt 0 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 5 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 12 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 12 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 12 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 5 client.4325.objecter 1 in flight 2023-12-01T17:06:00.165 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 16 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 13 ondisk uv 0 in 22.1 attempt 0 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 10 client.4325.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 13 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 13 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 13 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (326 ms) 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02db9d0 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 59 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:00.166 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [read 0~0] tid 14 osd.2 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 14 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 15 client.4325.objecter _send_op 14 to 22.1 on osd.2 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:14 22.1 22:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59) v8 -- 0x55e7c02dbe70 con 0x55e7c020b6e0 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02fff70 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 59 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.a7bbaa1 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.163+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.164+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid noexist '@22' '@22' [read 0~0] tid 15 osd.2 2023-12-01T17:06:00.167 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.164+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.271 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:00.216+0000 12256640 -1 osd.3 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:00.490 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.164+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 15 2023-12-01T17:06:00.490 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.164+0000 7f3f1da37780 15 client.4325.objecter _send_op 15 to 22.1 on osd.2 2023-12-01T17:06:00.490 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.164+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:15 22.1 22:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e59) v8 -- 0x55e7c0300410 con 0x55e7c020b6e0 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.164+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.164+0000 7f3f1da37780 5 client.4325.objecter 2 in flight 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 17 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 14 ondisk uv 4 in 22.1 attempt 0 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 5 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 14 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 14 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 14 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.361+0000 7f3f1b4e4640 5 client.4325.objecter 1 in flight 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 18 ==== osd_map(60..60 src has 1..61) v4 ==== 688+0+0 (crc 0 0 0) 0x7f3f10052ab0 con 0x55e7c020b6e0 2023-12-01T17:06:00.491 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(60..60 src has 1..61) v4 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 19 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [60,60] > 59 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 60 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 10 client.4325.objecter checking op 15 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 60 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f03fff640 20 client.4325.objecter 15 22.a7bbaa1 osd.2 noexist [read 0~0] 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 15 ondisk uv 0 in 22.1 attempt 0 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 10 client.4325.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-01T17:06:00.492 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 15 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 15 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 15 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (327 ms) 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02ff070 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 60 base exist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:00.493 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [read 0~0] tid 16 osd.2 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 16 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 15 client.4325.objecter _send_op 16 to 22.1 on osd.2 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:16 22.1 22:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) v8 -- 0x55e7c02ff510 con 0x55e7c020b6e0 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02ffa30 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 60 base noexist @22 precalc_pgid 0 pgid 0.0 is_read 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter _calc_target target noexist @22 -> pgid 22.a7bbaa1 2023-12-01T17:06:00.494 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.a7bbaa1 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:00.495 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.495 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid noexist '@22' '@22' [read 0~0] tid 17 osd.2 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 17 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 15 client.4325.objecter _send_op 17 to 22.1 on osd.2 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:17 22.1 22:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60) v8 -- 0x55e7c02fff50 con 0x55e7c020b6e0 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.490+0000 7f3f1da37780 5 client.4325.objecter 2 in flight 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (278 ms) 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2023-12-01T17:06:00.769 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 20 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 16 ondisk uv 4 in 22.1 attempt 0 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 5 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 16 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 16 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 5 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 16 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 5 client.4325.objecter 1 in flight 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 21 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 17 ondisk uv 0 in 22.1 attempt 0 2023-12-01T17:06:00.770 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 10 client.4325.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2023-12-01T17:06:00.771 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:06:00.771 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 17 2023-12-01T17:06:00.771 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 17 2023-12-01T17:06:00.771 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:00.771 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 17 2023-12-01T17:06:00.771 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:06:01.407 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:01.399+0000 12256640 -1 osd.3 61 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:01.457 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12 api_misc: [==========] Running 12 tests from 4 test suites. 2023-12-01T17:06:01.457 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] Global test environment set-up. 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: 2023-12-01T17:05:02.928+0000 7f1c57672780 -1 failed for service _ceph-mon._tcp 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: 2023-12-01T17:05:02.928+0000 7f1c57672780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (815 ms) 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2009 ms) 2023-12-01T17:06:01.458 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2824 ms total) 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2bb04e50 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2aeebca0 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b271950 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b2722e0 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b272cc0 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b2736c0 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b2740c0 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b274ac0 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b275470 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b1409d0 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b1d7a00 2023-12-01T17:06:01.459 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b26abe0 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b26ad90 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b049f30 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b2752a0 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04add0 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b049d30 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04bcb0 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04ac40 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04cc30 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04bab0 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04db20 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04c990 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04ea30 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04d8e0 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04f8e0 2023-12-01T17:06:01.460 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04e770 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b050770 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b04f650 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b0515e0 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b050500 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b052450 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b051390 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b052fe0 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b053a40 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b052200 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b0548c0 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b0537f0 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b055780 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b054680 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b056630 2023-12-01T17:06:01.461 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b0554e0 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b0574d0 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b056400 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b058390 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b057260 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b059200 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b058130 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b275fa0 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b276a10 2023-12-01T17:06:01.462 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b058fe0 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: starte-01T17:06:00.768+0000 7f3f1da37780 20 librados: queue_aio_write 0x55e7c010a480 completion 0x55e7c02c75d0 write_seq 4 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02ffa30 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 60 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [writefull 0~5 in=5b] tid 18 osd.2 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 18 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 15 client.4325.objecter _send_op 18 to 22.1 on osd.2 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:18 22.1 22:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) v8 -- 0x55e7c02db9d0 con 0x55e7c020b6e0 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:02.053 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:00.768+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:01.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:01.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:01.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 22 ==== osd_op_reply(18 exist [writefull 0~5] v60'5 uv5 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(18 exist [writefull 0~5] v60'5 uv5 ondisk = 0) v8 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 18 ondisk uv 5 in 22.1 attempt 0 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 18 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 18 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 18 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:06:02.054 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1b4e4640 20 librados: complete_aio_write 0x55e7c02c75d0 2023-12-01T17:06:02.055 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1284 ms) 2023-12-01T17:06:02.055 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2023-12-01T17:06:02.483 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:02.471+0000 12256640 -1 osd.3 62 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 20 librados: queue_aio_write 0x55e7c010a480 completion 0x55e7c02db8c0 write_seq 5 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02ffa30 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 60 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [writefull 0~5 in=5b] tid 19 osd.2 2023-12-01T17:06:02.585 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:02.586 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 19 2023-12-01T17:06:02.586 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 15 client.4325.objecter _send_op 19 to 22.1 on osd.2 2023-12-01T17:06:02.586 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:19 22.1 22:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) v8 -- 0x55e7c02db9d0 con 0x55e7c020b6e0 2023-12-01T17:06:02.586 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:02.586 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.052+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:06:02.586 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.145+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.145+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.145+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.270+0000 7f3f1cf47640 10 client.4325.objecter tick 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 23 ==== osd_map(61..61 src has 1..61) v4 ==== 304+0+0 (crc 0 0 0) 0x7f3f10052ab0 con 0x55e7c020b6e0 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(61..61 src has 1..61) v4 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [61,61] > 60 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 61 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 10 client.4325.objecter checking op 19 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 61 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.320+0000 7f3f03fff640 20 client.4325.objecter 19 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (532 ms) 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2023-12-01T17:06:02.587 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 24 ==== osd_op_reply(19 exist [writefull 0~5] v61'6 uv6 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(19 exist [writefull 0~5] v61'6 uv6 ondisk = 0) v8 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 19 ondisk uv 6 in 22.1 attempt 0 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 19 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 19 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 19 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1b4e4640 20 librados: complete_aio_write 0x55e7c02db8c0 2023-12-01T17:06:02.854 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 20 librados: queue_aio_write 0x55e7c010a480 completion 0x55e7c02dbec0 write_seq 6 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 10 client.4325.objecter _op_submit op 0x55e7c02ffa30 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 20 client.4325.objecter _calc_target epoch 61 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 10 client.4325.objecter _calc_target raw pgid 22.d3a59201 -> actual 22.1 acting [2,6] primary 2 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 20 client.4325.objecter _get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 10 client.4325.objecter _op_submit oid exist '@22' '@22' [writefull 0~5 in=5b] tid 20 osd.2 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 20 client.4325.objecter get_session s=0x55e7c00cf140 osd=2 3 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 15 client.4325.objecter _session_op_assign 2 20 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 15 client.4325.objecter _send_op 20 to 22.1 on osd.2 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- osd_op(unknown.0.0:20 22.1 22:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61) v8 -- 0x55e7c02db9d0 con 0x55e7c020b6e0 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.585+0000 7f3f1da37780 5 client.4325.objecter 1 in flight 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 25 ==== osd_map(62..62 src has 1..63) v4 ==== 668+0+0 (crc 0 0 0) 0x7f3f10052ab0 con 0x55e7c020b6e0 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_map(62..62 src has 1..63) v4 2023-12-01T17:06:02.855 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map got epochs [62,62] > 61 2023-12-01T17:06:02.856 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 3 client.4325.objecter handle_osd_map decoding incremental epoch 62 2023-12-01T17:06:02.856 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 10 client.4325.objecter checking op 20 2023-12-01T17:06:03.087 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:03.066+0000 11a55640 -1 osd.0 62 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:03.603 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:03.601+0000 12256640 -1 osd.3 63 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:04.274 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:04.228+0000 11a55640 -1 osd.0 63 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:04.976 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:04.917+0000 12256640 -1 osd.3 64 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 20 client.4325.objecter _calc_target epoch 62 base exist @22 precalc_pgid 0 pgid 0.0 is_write 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 20 client.4325.objecter _calc_target target exist @22 -> pgid 22.d3a59201 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 20 client.4325.objecter dump_active .. 0 homeless 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:02.853+0000 7f3f03fff640 20 client.4325.objecter 20 22.d3a59201 osd.2 exist [writefull 0~5 in=5b] 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:03.146+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:03.146+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:03.146+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:04.146+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:04.146+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:04.146+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.146+0000 7f3f02ffd640 10 monclient: tick 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.146+0000 7f3f02ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:06:05.157 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.146+0000 7f3f02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.146+0000 7f3f02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 <== osd.2 v2:172.21.15.67:6818/2342127620 26 ==== osd_op_reply(20 exist [writefull 0~5] v62'7 uv7 ondisk = 0) v8 ==== 149+0+0 (crc 0 0 0) 0x7f3f10059040 con 0x55e7c020b6e0 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 10 client.4325.objecter ms_dispatch 0x55e7c01041b0 osd_op_reply(20 exist [writefull 0~5] v62'7 uv7 ondisk = 0) v8 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 10 client.4325.objecter in handle_osd_op_reply 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 7 client.4325.objecter handle_osd_op_reply 20 ondisk uv 7 in 22.1 attempt 0 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 10 client.4325.objecter op 0 rval 0 len 0 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 15 client.4325.objecter handle_osd_op_reply completed tid 20 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 15 client.4325.objecter _finish_op 20 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 4 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 15 client.4325.objecter _session_op_remove 2 20 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 5 client.4325.objecter 0 in flight 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1b4e4640 20 librados: complete_aio_write 0x55e7c02dbec0 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2572 ms) 2023-12-01T17:06:05.158 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 10 librados: watch_flush enter 2023-12-01T17:06:05.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 10 librados: watch_flush exit 2023-12-01T17:06:05.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 10 client.4325.objecter close_session for osd.2 2023-12-01T17:06:05.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 msgr2=0x55e7c02db380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:06:05.159 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x55e7c020b6e0 0x55e7c02db380 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 20 client.4325.objecter put_session s=0x55e7c00cf140 osd=2 1 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 20 client.4325.objecter shutdown clearing up homeless session... 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.156+0000 7f3f1da37780 10 client.4325.objecter successfully canceled tick 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3f0806aa70 msgr2=0x7f3f0806cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f3f0806aa70 0x7f3f0806cf30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3f0c0096e0 tx=0x7f3f0c00c040 comp rx=0 tx=0).stop 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 10 monclient: shutdown 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 msgr2=0x55e7c02c6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 --2- 172.21.15.67:0/1986628037 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x55e7c0167fa0 0x55e7c02c6350 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3f14005fd0 tx=0x7f3f140084f0 comp rx=0 tx=0).stop 2023-12-01T17:06:05.161 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1b4e4640 1 -- 172.21.15.67:0/1986628037 reap_dead start 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 shutdown_connections 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 >> 172.21.15.67:0/1986628037 conn(0x55e7c0198020 msgr2=0x55e7c01637d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 shutdown_connections 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 -- 172.21.15.67:0/1986628037 wait complete. 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.157+0000 7f3f1da37780 1 librados: shutdown 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [----------] 12 tests from AsioRados (20226 ms total) 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [----------] Global test environment tear-down 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (63060 ms total) 2023-12-01T17:06:05.162 INFO:tasks.workunit.client.0.smithi067.stdout: api_asio: [ PASSED ] 12 tests. 2023-12-01T17:06:05.507 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:05.385+0000 11a55640 -1 osd.0 64 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:05.962 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:05.961+0000 12256640 -1 osd.3 65 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:06.526 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:06.411+0000 11a55640 -1 osd.0 65 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:07.001 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:06.943+0000 12256640 -1 osd.3 65 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:07.685 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: Running main() from gmock_main.cc 2023-12-01T17:06:07.685 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2023-12-01T17:06:07.685 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [----------] Global test environment set-up. 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: handle_notify cookie 93864183490176 notify_id 167503724544 notifier_gid 4549 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (32511 ms) 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: Trying... 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: handle_notify cookie 93864196070480 notify_id 261993005056 notifier_gid 4783 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: Waiting for 3.000000000s 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: Timed out. 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: Flushing... 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: Flushed... 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (32673 ms) 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (65184 ms total) 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: 2023-12-01T17:06:07.686 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [----------] Global test environment tear-down 2023-12-01T17:06:07.687 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (65184 ms total) 2023-12-01T17:06:07.687 INFO:tasks.workunit.client.0.smithi067.stdout: watch_notify: [ PASSED ] 2 tests. 2023-12-01T17:06:08.036 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:07.951+0000 12256640 -1 osd.3 66 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:08.040 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:07.797+0000 11a55640 -1 osd.0 67 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:08.983 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:08.981+0000 11a55640 -1 osd.0 67 get_health_metrics reporting 26 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:09.170 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: Running main() from gmock_main.cc 2023-12-01T17:06:09.170 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2023-12-01T17:06:09.170 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [----------] Global test environment set-up. 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [----------] 9 tests from LibRadosPools 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolList (8514 ms) 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (5821 ms) 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (5732 ms) 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (8923 ms) 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (8506 ms) 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (6483 ms) 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5897 ms) 2023-12-01T17:06:09.171 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5888 ms) 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (11258 ms) 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [----------] 9 tests from LibRadosPools (67022 ms total) 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [----------] Global test environment tear-down 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (67022 ms total) 2023-12-01T17:06:09.172 INFO:tasks.workunit.client.0.smithi067.stdout: api_pool: [ PASSED ] 9 tests. 2023-12-01T17:06:09.190 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:09.074+0000 12256640 -1 osd.3 67 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:09.318 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:23.099 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.327 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:23.108 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.393 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:23.153 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.400 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:23.160 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.417 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:23.198 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.429 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:23.210 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.473 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:06:09.374+0000 11254640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4535.0:2 59.15 59.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-01T17:06:09.520 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:23.280 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.543 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:23.324 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:09.589 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:23.369 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:10.340 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:10.196+0000 12256640 -1 osd.3 68 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:10.492 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:10.467+0000 11a55640 -1 osd.0 68 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4397.0:23 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40) 2023-12-01T17:06:10.637 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:06:10.515+0000 11254640 -1 osd.2 67 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4535.0:2 59.15 59.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-01T17:06:11.239 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:11.219+0000 12256640 -1 osd.3 68 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:11.878 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:06:11.865+0000 11254640 -1 osd.2 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4535.0:2 59.15 59.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-01T17:06:12.102 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:12.096+0000 11a55640 -1 osd.0 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4397.0:23 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40) 2023-12-01T17:06:12.636 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:12.632+0000 12256640 -1 osd.3 69 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:13.008 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:06:13.006+0000 11254640 -1 osd.2 69 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4535.0:2 59.15 59.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-01T17:06:13.373 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:13.252+0000 11a55640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4397.0:23 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40) 2023-12-01T17:06:13.885 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:13.878+0000 12256640 -1 osd.3 70 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:14.311 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:14.290+0000 11a55640 -1 osd.0 71 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4397.0:23 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40) 2023-12-01T17:06:14.697 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:06:14.613+0000 11254640 -1 osd.2 70 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4535.0:2 59.15 59.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-01T17:06:14.915 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:14.892+0000 12256640 -1 osd.3 71 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:15.426 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:15.415+0000 11a55640 -1 osd.0 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4397.0:23 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40) 2023-12-01T17:06:15.932 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:15.925+0000 12256640 -1 osd.3 72 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:16.336 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:06:16.214+0000 11254640 -1 osd.2 72 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4535.0:2 59.15 59.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-01T17:06:16.551 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:16.438+0000 11a55640 -1 osd.0 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4397.0:23 38.17 38.b262ca77 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e40) 2023-12-01T17:06:17.067 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:17.064+0000 12256640 -1 osd.3 73 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:17.522 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:06:17.479+0000 11254640 -1 osd.2 73 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4535.0:2 59.15 59.40e8aab5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e41) 2023-12-01T17:06:18.317 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:18.284+0000 12256640 -1 osd.3 74 get_health_metrics reporting 4 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:18.738 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:18.588+0000 11a55640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:19.360 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:19.268+0000 12256640 -1 osd.3 75 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:19.829 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:19.827+0000 11a55640 -1 osd.0 75 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:20.303 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:20.298+0000 12256640 -1 osd.3 75 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:21.506 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:21.452+0000 11a55640 -1 osd.0 77 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:21.920 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:21.782+0000 12256640 -1 osd.3 78 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:22.590 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:22.548+0000 11a55640 -1 osd.0 78 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:22.910 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:22.893+0000 12256640 -1 osd.3 80 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:23.421 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:06:23.273+0000 10a53640 -1 osd.5 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:5 7.8 7:13d2f738:test-rados-api-smithi067-92541-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2023-12-01T17:06:23.809 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:23.803+0000 11a55640 -1 osd.0 79 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:24.014 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:24.000+0000 12256640 -1 osd.3 80 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:24.516 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:06:24.514+0000 10a53640 -1 osd.5 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:5 7.8 7:13d2f738:test-rados-api-smithi067-92541-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2023-12-01T17:06:24.917 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:24.905+0000 11a55640 -1 osd.0 80 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:25.119 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:25.102+0000 12256640 -1 osd.3 81 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:25.593 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:06:25.583+0000 10a53640 -1 osd.5 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:5 7.8 7:13d2f738:test-rados-api-smithi067-92541-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2023-12-01T17:06:25.936 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:25.930+0000 11a55640 -1 osd.0 81 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:26.153 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:26.150+0000 12256640 -1 osd.3 82 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:26.561 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:06:26.560+0000 10a53640 -1 osd.5 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:5 7.8 7:13d2f738:test-rados-api-smithi067-92541-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2023-12-01T17:06:26.880 INFO:tasks.workunit.client.0.smithi067.stdout:d 0x564d2b277890 2023-12-01T17:06:26.880 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b2767c0 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b278750 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b277640 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b2795f0 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b2784b0 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27a480 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b279380 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27b310 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27a230 2023-12-01T17:06:26.881 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27c190 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27b0d0 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27cd20 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27d790 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27bf20 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27e280 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27ebe0 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 0x564d2b27d520 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: started 69 aios 2023-12-01T17:06:26.882 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2bb04e50 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2aeebca0 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b271950 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b2722e0 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b272cc0 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b2736c0 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b2740c0 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b274ac0 2023-12-01T17:06:26.883 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b275470 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b1409d0 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b1d7a00 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b26abe0 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b26ad90 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b049f30 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b2752a0 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04add0 2023-12-01T17:06:26.884 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b049d30 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04bcb0 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04ac40 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04cc30 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04bab0 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04db20 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04c990 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04ea30 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04d8e0 2023-12-01T17:06:26.885 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04f8e0 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04e770 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b050770 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b04f650 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b0515e0 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b050500 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b052450 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b051390 2023-12-01T17:06:26.886 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b052fe0 2023-12-01T17:06:26.887 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b053a40 2023-12-01T17:06:26.887 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b052200 2023-12-01T17:06:26.887 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b0548c0 2023-12-01T17:06:26.887 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b0537f0 2023-12-01T17:06:26.887 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b055780 2023-12-01T17:06:26.887 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b054680 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b056630 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b0554e0 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b0574d0 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b056400 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b058390 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b057260 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b059200 2023-12-01T17:06:26.888 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b058130 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b275fa0 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b276a10 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b058fe0 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b277890 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b2767c0 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b278750 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b277640 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b2795f0 2023-12-01T17:06:26.889 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b2784b0 2023-12-01T17:06:26.890 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27a480 2023-12-01T17:06:26.890 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b279380 2023-12-01T17:06:26.890 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27b310 2023-12-01T17:06:26.890 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27a230 2023-12-01T17:06:26.890 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27c190 2023-12-01T17:06:26.890 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27b0d0 2023-12-01T17:06:27.121 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:27.115+0000 11a55640 -1 osd.0 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:27.149 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:27.142+0000 12256640 -1 osd.3 82 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:27.602 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:06:27.601+0000 10a53640 -1 osd.5 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:5 7.8 7:13d2f738:test-rados-api-smithi067-92541-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2023-12-01T17:06:28.169 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:28.160+0000 12256640 -1 osd.3 82 get_health_metrics reporting 6 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:28.171 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:28.162+0000 11a55640 -1 osd.0 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:29.113 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:29.112+0000 12256640 -1 osd.3 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:29.217 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:29.203+0000 11a55640 -1 osd.0 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:29.646 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:06:29.645+0000 10a53640 -1 osd.5 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:5 7.8 7:13d2f738:test-rados-api-smithi067-92541-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2023-12-01T17:06:30.180 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:43.961 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.189 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:43.970 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.252 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:44.011 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.252 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:44.013 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.253 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:44.034 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.269 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:44.050 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.339 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:03:44.100 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.369 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:03:44.150 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.408 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:03:44.189 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:30.540 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:06:30.351+0000 11a55640 -1 osd.0 82 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4483.0:2 12.0 12:07405f84:test-rados-api-smithi067-92416-3::aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:head [write 0~7 in=7b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:30.543 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:30.437+0000 12256640 -1 osd.3 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:30.917 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:06:30.819+0000 10a53640 -1 osd.5 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4477.0:5 7.8 7:13d2f738:test-rados-api-smithi067-92541-3::foo:head [write 0~128 in=128b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) 2023-12-01T17:06:31.498 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:31.482+0000 12256640 -1 osd.3 83 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:32.718 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:32.713+0000 12256640 -1 osd.3 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31.71d521c0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:34.004 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:33.923+0000 12256640 -1 osd.3 84 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:34.974 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:34.972+0000 12256640 -1 osd.3 85 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4579.0:2 31.0 31:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:36.487 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:36.409+0000 12256640 -1 osd.3 86 get_health_metrics reporting 5 slow ops, oldest is osd_op(client.4579.0:2 31.0 31:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:37.625 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:37.534+0000 12256640 -1 osd.3 87 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:39.400 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:39.217+0000 12256640 -1 osd.3 88 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:41.666 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:41.647+0000 12256640 -1 osd.3 89 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4579.0:2 31.0 31:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:42.687 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:06:42.683+0000 12256640 -1 osd.3 89 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4579.0:2 31.0 31:0384ab8e:::oid:head [write 18446744073709551615~0] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24) 2023-12-01T17:06:49.696 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:04:03.476 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.705 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:04:03.486 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.777 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:04:03.558 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.779 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:04:03.560 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.781 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:04:03.542 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.785 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:04:03.545 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.857 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:04:03.637 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.901 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:04:03.661 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:06:49.909 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:04:03.690 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:07:02.858 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:02.787+0000 10a53640 -1 osd.1 107 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:02.909 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:07:02.822+0000 11a55640 -1 osd.0 109 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4437.0:10 19.18 19:1f402edb:::8.obj:head [write 0~23 in=23b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e77) 2023-12-01T17:07:04.796 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:04.697+0000 10a53640 -1 osd.1 109 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:06.210 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:06.080+0000 10a53640 -1 osd.1 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:07.610 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:07.597+0000 10a53640 -1 osd.1 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:08.969 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:08.956+0000 10a53640 -1 osd.1 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:10.490 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:10.368+0000 10a53640 -1 osd.1 112 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:11.888 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:11.754+0000 10a53640 -1 osd.1 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:13.212 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:12.966+0000 10a53640 -1 osd.1 113 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:14.445 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:14.370+0000 10a53640 -1 osd.1 114 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:15.764 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:07:15.536+0000 10a53640 -1 osd.1 116 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4486.0:7 9.10 9:0a98ba6d:test-rados-api-smithi067-92349-4::foo:head [call lock.lock in=55b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75) 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (150026 ms) 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (150026 ms total) 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (150026 ms total) 2023-12-01T17:07:32.386 INFO:tasks.workunit.client.0.smithi067.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2023-12-01T17:07:36.266 INFO:tasks.workunit.client.0.smithi067.stdout: cls: Running main() from gmock_main.cc 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [==========] Running 2 tests from 1 test suite. 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [----------] Global test environment set-up. 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [----------] 2 tests from NeoRadosCls 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [ RUN ] NeoRadosCls.DNE 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [ OK ] NeoRadosCls.DNE (36169 ms) 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (117718 ms) 2023-12-01T17:07:36.267 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [----------] 2 tests from NeoRadosCls (153887 ms total) 2023-12-01T17:07:36.268 INFO:tasks.workunit.client.0.smithi067.stdout: cls: 2023-12-01T17:07:36.268 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [----------] Global test environment tear-down 2023-12-01T17:07:36.268 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [==========] 2 tests from 1 test suite ran. (153888 ms total) 2023-12-01T17:07:36.268 INFO:tasks.workunit.client.0.smithi067.stdout: cls: [ PASSED ] 2 tests. 2023-12-01T17:07:50.811 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: Running main() from gmock_main.cc 2023-12-01T17:07:50.811 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2023-12-01T17:07:50.811 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [----------] Global test environment set-up. 2023-12-01T17:07:50.811 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2023-12-01T17:07:50.811 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2023-12-01T17:07:50.811 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (801 ms) 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (155 ms) 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (167583 ms) 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (168539 ms total) 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [----------] Global test environment tear-down 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (168540 ms total) 2023-12-01T17:07:50.812 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2023-12-01T17:07:51.893 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: Running main() from gmock_main.cc 2023-12-01T17:07:51.893 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [----------] Global test environment set-up. 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (5383 ms) 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (20114 ms) 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (143972 ms) 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [----------] 3 tests from NeoRadosCmd (169469 ms total) 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [----------] Global test environment tear-down 2023-12-01T17:07:51.894 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [==========] 3 tests from 1 test suite ran. (169469 ms total) 2023-12-01T17:07:51.895 INFO:tasks.workunit.client.0.smithi067.stdout: cmd: [ PASSED ] 3 tests. 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] Global test environment set-up. 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] 7 tests from LibRadosList 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosList.ListObjects (143358 ms) 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (9929 ms) 2023-12-01T17:07:53.214 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset foo1,foo2,foo3 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo1 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo2 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo3 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset foo1,foo4,foo5 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo4 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo5 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo1 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset foo6,foo7 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo7 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo6 2023-12-01T17:07:53.215 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns1:foo4 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns1:foo5 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns2:foo7 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns2:foo6 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns1:foo1 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: :foo1 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: :foo2 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: :foo3 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (5366 ms) 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 1 0 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 10 0 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 13 0 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 7 0 2023-12-01T17:07:53.216 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 14 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 0 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 15 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 11 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 5 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 8 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 6 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 3 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 4 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 12 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 9 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 2 0 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (2010 ms) 2023-12-01T17:07:53.217 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: x cursor=MIN 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=1 cursor=15:02547ec2:::1:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=10 cursor=15:52ea6a34:::10:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=13 cursor=15:566253c9:::13:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=7 cursor=15:5c6b0b28:::7:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=14 cursor=15:62a1935d:::14:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=0 cursor=15:6cac518f:::0:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=15 cursor=15:863748b0:::15:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=11 cursor=15:89d3ae78:::11:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=5 cursor=15:b29083e3:::5:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=8 cursor=15:bd63b0f1:::8:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=6 cursor=15:c4fdafeb:::6:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=3 cursor=15:cfc208b3:::3:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=4 cursor=15:d83876eb:::4:head 2023-12-01T17:07:53.218 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=12 cursor=15:de5d7c5f:::12:head 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=9 cursor=15:e960b815:::9:head 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > oid=2 cursor=15:f905c69b:::2:head 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: FIRST> seek to MIN oid=1 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=1 cursor=15:02547ec2:::1:head 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:02547ec2:::1:head 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:02547ec2:::1:head -> 1 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=10 cursor=15:52ea6a34:::10:head 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:52ea6a34:::10:head 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:52ea6a34:::10:head -> 10 2023-12-01T17:07:53.219 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=13 cursor=15:566253c9:::13:head 2023-12-01T17:07:53.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:566253c9:::13:head 2023-12-01T17:07:53.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:566253c9:::13:head -> 13 2023-12-01T17:07:53.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=7 cursor=15:5c6b0b28:::7:head 2023-12-01T17:07:53.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:5c6b0b28:::7:head 2023-12-01T17:07:53.220 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:5c6b0b28:::7:head -> 7 2023-12-01T17:07:55.259 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : o api_c_read_operations: Running main() from gmock_main.cc 2023-12-01T17:07:55.259 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [----------] Global test environment set-up. 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (17901 ms) 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (106507 ms) 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (4538 ms) 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (16196 ms) 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2023-12-01T17:07:55.260 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (138 ms) 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (522 ms) 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (3200 ms) 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (358 ms) 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (2065 ms) 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (2597 ms) 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (112 ms) 2023-12-01T17:07:55.261 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (97 ms) 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (1644 ms) 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (472 ms) 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (409 ms) 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (1715 ms) 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (158471 ms total) 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [----------] Global test environment tear-down 2023-12-01T17:07:55.262 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (172970 ms total) 2023-12-01T17:07:55.263 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2023-12-01T17:07:58.916 INFO:tasks.workunit.client.0.smithi067.stdout:id=14 cursor=15:62a1935d:::14:head 2023-12-01T17:07:58.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:62a1935d:::14:head 2023-12-01T17:07:58.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:62a1935d:::14:head -> 14 2023-12-01T17:07:58.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=0 cursor=15:6cac518f:::0:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:6cac518f:::0:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:6cac518f:::0:head -> 0 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=15 cursor=15:863748b0:::15:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:863748b0:::15:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:863748b0:::15:head -> 15 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=11 cursor=15:89d3ae78:::11:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:89d3ae78:::11:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:89d3ae78:::11:head -> 11 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=5 cursor=15:b29083e3:::5:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:b29083e3:::5:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:b29083e3:::5:head -> 5 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=8 cursor=15:bd63b0f1:::8:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:bd63b0f1:::8:head 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:bd63b0f1:::8:head -> 8 2023-12-01T17:07:58.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=6 cursor=15:c4fdafeb:::6:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:c4fdafeb:::6:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:c4fdafeb:::6:head -> 6 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=3 cursor=15:cfc208b3:::3:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:cfc208b3:::3:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:cfc208b3:::3:head -> 3 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=4 cursor=15:d83876eb:::4:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:d83876eb:::4:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:d83876eb:::4:head -> 4 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=12 cursor=15:de5d7c5f:::12:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:de5d7c5f:::12:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:de5d7c5f:::12:head -> 12 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=9 cursor=15:e960b815:::9:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:e960b815:::9:head 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:e960b815:::9:head -> 9 2023-12-01T17:07:58.918 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : oid=2 cursor=15:f905c69b:::2:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:f905c69b:::2:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:f905c69b:::2:head -> 2 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:6cac518f:::0:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:6cac518f:::0:head expected=15:6cac518f:::0:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:6cac518f:::0:head -> 0 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=0 expected=0 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:62a1935d:::14:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:62a1935d:::14:head expected=15:62a1935d:::14:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:62a1935d:::14:head -> 14 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=14 expected=14 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:5c6b0b28:::7:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:5c6b0b28:::7:head expected=15:5c6b0b28:::7:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:5c6b0b28:::7:head -> 7 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=7 expected=7 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:566253c9:::13:head 2023-12-01T17:07:58.919 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:566253c9:::13:head expected=15:566253c9:::13:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:566253c9:::13:head -> 13 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=13 expected=13 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:52ea6a34:::10:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:52ea6a34:::10:head expected=15:52ea6a34:::10:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:52ea6a34:::10:head -> 10 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=10 expected=10 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:02547ec2:::1:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:02547ec2:::1:head expected=15:02547ec2:::1:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:02547ec2:::1:head -> 1 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=1 expected=1 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:f905c69b:::2:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:f905c69b:::2:head expected=15:f905c69b:::2:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:f905c69b:::2:head -> 2 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=2 expected=2 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:e960b815:::9:head 2023-12-01T17:07:58.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:e960b815:::9:head expected=15:e960b815:::9:head 2023-12-01T17:07:58.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:e960b815:::9:head -> 9 2023-12-01T17:08:01.026 INFO:tasks.workunit.client.0.smithi067.stdout: api_ api_cmd: Running main() from gmock_main.cc 2023-12-01T17:08:01.026 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2023-12-01T17:08:01.026 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [----------] Global test environment set-up. 2023-12-01T17:08:01.026 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2023-12-01T17:08:01.026 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (1288 ms) 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (72 ms) 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (171903 ms) 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:43.104291+0000 mon.b [INF] from='client.? 172.21.15.67:0/1467921888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi067-92541-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi067-92541-7"}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:43.150774+0000 mon.b [INF] from='client.? 172.21.15.67:0/275545010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi067-92269-22"}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:43.252453+0000 mon.b [INF] from='client.? 172.21.15.67:0/106800212' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi067-92349-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:43.293830+0000 mon.b [INF] from='client.? 172.21.15.67:0/3813689178' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi067-92400-10"}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:45.236696+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi067-92541-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi067-92541-7"}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:45.243479+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi067-92269-22"}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:45.267933+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi067-92349-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:45.275349+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi067-92400-10"}]: dispatch 2023-12-01T17:08:01.027 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:45.409827+0000 mon.a [WRN] Health check update: 11 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-01T17:08:01.028 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:49.651894+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi067-92615-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-01T17:08:01.028 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:46.784482+0000 mon.c [INF] from='client.? 172.21.15.67:0/1454898481' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi067-92252-16"}]: dispatch 2023-12-01T17:08:01.028 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:50.965209+0000 mon.b [INF] from='client.? 172.21.15.67:0/275545010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi067-92269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:06.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:51.160563+0000 mon.b [INF] from='client.? 172.21.15.67:0/3813689178' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi067-92400-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:51.486027+0000 mon.c [INF] from='client.? 172.21.15.67:0/138728895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi067-92615-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi067-92615-7"}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:51.601296+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi067-92252-16"}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:52.866411+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi067-92269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:52.873356+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi067-92400-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:52.989217+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi067-92615-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi067-92615-7"}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:53.805683+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi067-92349-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:55.043276+0000 mon.b [INF] from='client.? 172.21.15.67:0/106800212' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi067-92349-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi067-92349-10"}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:55.205098+0000 mon.c [INF] from='client.? 172.21.15.67:0/1454898481' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi067-92252-16"}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:59.047532+0000 client.admin [INF] onexx 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:59.067800+0000 mon.a [INF] from='client.? 172.21.15.67:0/2317359396' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:59.496378+0000 mon.a [WRN] Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:07:59.643781+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi067-92349-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi067-92349-10"}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:00.283944+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi067-92252-16"}]: dispatch 2023-12-01T17:08:06.705 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:00.333667+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi067-92541-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi067-92541-7"}]': finished 2023-12-01T17:08:06.706 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:00.337201+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi067-92269-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-01T17:08:18.861 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:00.340685+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi067-92400-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:05.266947+0000 mon.b [INF] from='client.? 172.21.15.67:0/504867501' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi067-92243-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:05.311078+0000 mon.b [INF] from='client.? 172.21.15.67:0/3702829739' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi067-92226-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:07.309712+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi067-92243-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:07.316701+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi067-92226-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:08.644126+0000 mon.c [INF] from='client.? 172.21.15.67:0/3481901494' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi067-92662-12"}]: dispatch 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:08.813221+0000 client.admin [INF] twoxx 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:08.838266+0000 mon.a [INF] from='client.? 172.21.15.67:0/2317359396' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2023-12-01T17:08:18.862 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:09.463999+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi067-92662-12"}]: dispatch 2023-12-01T17:08:18.863 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:09.640965+0000 mon.b [INF] from='client.? 172.21.15.67:0/2833876847' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi067-92563-10"}]: dispatch 2023-12-01T17:08:18.863 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:09.997081+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi067-92349-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi067-92349-10"}]': finished 2023-12-01T17:08:18.863 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:10.000648+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi067-92252-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2023-12-01T17:08:18.863 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:09.410702+0000 mon.c [INF] from='client.? 172.21.15.67:0/4026042359' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi067-92859-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-01T17:08:18.863 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:10.985693+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi067-92563-10"}]: dispatch 2023-12-01T17:08:18.863 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:11.059054+0000 mon.c [INF] from='client.? 172.21.15.67:0/1454898481' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi067-92252-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi067-92252-16"}]: dispatch 2023-12-01T17:08:18.863 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:11.576539+0000 mon.c [INF] from='client.? 172.21.15.67:0/3481901494' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi067-92662-12"}]: dispatch 2023-12-01T17:08:19.480 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:05:33.261 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:19.496 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:05:33.277 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:20.173 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:05:33.953 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:20.363 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:34.123 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:20.398 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:34.156 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:20.577 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:34.337 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:20.664 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:05:34.445 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:20.698 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:05:34.479 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:21.048 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:05:34.826 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:26.206 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_1_[93066]: starting. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_1_[93066]: creating pool ceph_test_rados_open_pools_parallel.smithi067-93022 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_1_[93066]: created object 0... 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_1_[93066]: created object 25... 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_1_[93066]: created object 49... 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_1_[93066]: finishing. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_1_[93066]: shutting down. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_2_[93067]: starting. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_2_[93067]: rados_pool_create. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_2_[93067]: rados_ioctx_create. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_2_[93067]: shutting down. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: ******************************* 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_3_[134401]: starting. 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_3_[134401]: creating pool ceph_test_rados_open_pools_parallel.smithi067-93022 2023-12-01T17:08:26.207 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_3_[134401]: created object 0... 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_3_[134401]: created object 25... 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_3_[134401]: created object 49... 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_3_[134401]: finishing. 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_3_[134401]: shutting down. 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: ******************************* 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_4_[134402]: starting. 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_4_[134402]: rados_pool_create. 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_4_[134402]: rados_ioctx_create. 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: process_4_[134402]: shutting down. 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: ******************************* 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: ******************************* 2023-12-01T17:08:26.208 INFO:tasks.workunit.client.0.smithi067.stdout: open_pools_parallel: ******* SUCCESS ********** 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout:0x564d2b27cd20 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27d790 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27bf20 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27e280 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27ebe0 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: waiting 0x564d2b27d520 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: done. 2023-12-01T17:08:26.851 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (86561 ms) 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (86561 ms total) 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.Exec (11925 ms) 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (6087 ms) 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (9212 ms) 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.Applications (5367 ms) 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2023-12-01T17:08:26.852 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (79036 ms) 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] 8 tests from LibRadosMisc (111627 ms total) 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [----------] Global test environment tear-down 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (204729 ms total) 2023-12-01T17:08:26.853 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc: [ PASSED ] 12 tests. 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:11.992190+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi067-92859-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:13.579329+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi067-92269-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi067-92269-22"}]': finished 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:13.584063+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi067-92400-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi067-92400-10"}]': finished 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:13.586824+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi067-92243-7","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:13.588769+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi067-92226-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:13.636712+0000 mon.a [INF] from='client.? 172.21.15.67:0/2317359396' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:14.030380+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi067-92252-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi067-92252-16"}]: dispatch 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:14.077578+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi067-92662-12"}]: dispatch 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:15.221673+0000 mon.b [INF] from='client.? 172.21.15.67:0/2833876847' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi067-92563-10"}]: dispatch 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:15.441823+0000 mon.c [INF] from='client.? 172.21.15.67:0/3481901494' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi067-92662-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:16.229007+0000 mon.a [WRN] Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-12-01T17:08:27.730 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:17.166166+0000 client.admin [INF] threexx 2023-12-01T17:08:27.731 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:17.174361+0000 mon.a [INF] from='client.? 172.21.15.67:0/2317359396' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2023-12-01T17:08:27.731 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:17.358820+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi067-92563-10"}]: dispatch 2023-12-01T17:08:27.731 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:17.739917+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi067-92859-6","app": "rados","yes_i_really_mean_it": true}]': finished 2023-12-01T17:08:27.731 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: got: 2023-12-01T17:08:18.487716+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi067-92662-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2023-12-01T17:08:27.731 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (32219 ms) 2023-12-01T17:08:27.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (205482 ms total) 2023-12-01T17:08:27.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: 2023-12-01T17:08:27.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [----------] Global test environment tear-down 2023-12-01T17:08:27.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (205484 ms total) 2023-12-01T17:08:27.732 INFO:tasks.workunit.client.0.smithi067.stdout: api_cmd: [ PASSED ] 4 tests. 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: Running main() from gmock_main.cc 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [----------] Global test environment set-up. 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [----------] 5 tests from LibRadosStat 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStat.Stat (45014 ms) 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (40059 ms) 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStat.StatNS (23334 ms) 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2023-12-01T17:08:28.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (4 ms) 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (19975 ms) 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [----------] 5 tests from LibRadosStat (128387 ms total) 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (2675 ms) 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (883 ms) 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1306 ms) 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2023-12-01T17:08:28.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (4508 ms) 2023-12-01T17:08:28.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (9372 ms total) 2023-12-01T17:08:28.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: 2023-12-01T17:08:28.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [----------] Global test environment tear-down 2023-12-01T17:08:28.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (205890 ms total) 2023-12-01T17:08:28.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat: [ PASSED ] 9 tests. 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: Running main() from gmock_main.cc 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [----------] Global test environment set-up. 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: seed 92615 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (29966 ms) 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (7485 ms) 2023-12-01T17:08:28.072 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (27 ms) 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (2647 ms) 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (95451 ms) 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (135576 ms total) 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (6820 ms) 2023-12-01T17:08:28.073 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1986 ms) 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2164 ms) 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (282 ms) 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (11253 ms total) 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [----------] Global test environment tear-down 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (205897 ms total) 2023-12-01T17:08:28.074 INFO:tasks.workunit.client.0.smithi067.stdout: api_stat_pp: [ PASSED ] 9 tests. 2023-12-01T17:08:41.890 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:05:55.671 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:41.894 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:05:55.675 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:41.952 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:55.712 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:41.954 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:55.714 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:41.988 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:05:55.768 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:42.009 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:05:55.790 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:42.039 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:05:55.819 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:42.048 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:05:55.808 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:42.143 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:05:55.924 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:08:50.407 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: Running main() from gmock_main.cc 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [----------] Global test environment set-up. 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: seed 92269 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (62641 ms) 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (3086 ms) 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2023-12-01T17:08:50.408 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (2595 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (11285 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (5680 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2245 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (2020 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5936 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (8763 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5586 ms) 2023-12-01T17:08:50.409 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (9539 ms) 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (3223 ms) 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2352 ms) 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (962 ms) 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (5731 ms) 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (1202 ms) 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (1612 ms) 2023-12-01T17:08:50.410 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (1634 ms) 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (3105 ms) 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (139197 ms total) 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (11331 ms) 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (14432 ms) 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (2275 ms) 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2023-12-01T17:08:50.411 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (1975 ms) 2023-12-01T17:08:50.412 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2023-12-01T17:08:50.822 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRado api_lock: Running main() from gmock_main.cc 2023-12-01T17:08:50.822 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [----------] Global test environment set-up. 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [----------] 8 tests from LibRadosLock 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (56490 ms) 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.LockShared (16238 ms) 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (57052 ms) 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1934 ms) 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (3202 ms) 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3625 ms) 2023-12-01T17:08:50.823 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (116 ms) 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (144 ms) 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [----------] 8 tests from LibRadosLock (138801 ms total) 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (6966 ms) 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (293 ms) 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1375 ms) 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (6904 ms) 2023-12-01T17:08:50.824 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (10079 ms) 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (3006 ms) 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (2839 ms) 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (2284 ms) 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (33746 ms total) 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [----------] Global test environment tear-down 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (228765 ms total) 2023-12-01T17:08:50.825 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock: [ PASSED ] 16 tests. 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: Running main() from gmock_main.cc 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [----------] Global test environment set-up. 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify_test_cb 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (127568 ms) 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94256716013872 err -107 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (5859 ms) 2023-12-01T17:08:58.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94256716015968 err -107 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (2711 ms) 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 523986010112 cookie 94256716032944 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (4412 ms) 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 528280977409 cookie 94256716032944 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (3528 ms) 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 532575944704 cookie 94256716032944 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 532575944705 cookie 94256716032944 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (6307 ms) 2023-12-01T17:08:58.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 536870912000 cookie 94256716050864 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 536870912000 cookie 94256716055088 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 536870912001 cookie 94256716050864 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 536870912001 cookie 94256716055088 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (437 ms) 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 541165879296 cookie 94256716055088 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 541165879297 cookie 94256716055088 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (5987 ms) 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94256716055088 err -107 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_cb from 4576 notify_id 545460846594 cookie 94256716055088 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5274 ms) 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2023-12-01T17:08:58.101 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2023-12-01T17:08:58.102 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94256716055088 err -107 2023-12-01T17:08:58.102 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (2582 ms) 2023-12-01T17:08:58.102 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (164665 ms total) 2023-12-01T17:08:58.102 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: 2023-12-01T17:08:58.102 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2023-12-01T17:08:58.102 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2023-12-01T17:08:58.102 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: watch_notify_test_cb 2023-12-01T17:09:01.440 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (31193 ms) 2023-12-01T17:09:01.440 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (31193 ms total) 2023-12-01T17:09:01.440 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: 2023-12-01T17:09:01.440 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [----------] Global test environment tear-down 2023-12-01T17:09:01.440 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (239239 ms total) 2023-12-01T17:09:01.440 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify: [ PASSED ] 11 tests. 2023-12-01T17:09:01.841 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:15.622 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:01.851 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:15.632 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:01.908 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:15.667 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:01.908 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:15.669 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:01.919 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:15.700 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:01.922 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:15.703 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:01.954 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:15.735 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:02.002 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:15.762 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:02.023 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:15.804 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:13.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: Running main() from gmock_main.cc 2023-12-01T17:09:13.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2023-12-01T17:09:13.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [----------] Global test environment set-up. 2023-12-01T17:09:13.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2023-12-01T17:09:13.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: seed 92400 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (22483 ms) 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (765 ms) 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (92287 ms) 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (4813 ms) 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (11722 ms) 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (98 ms) 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (2358 ms) 2023-12-01T17:09:13.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (5215 ms) 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (139741 ms total) 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (6921 ms) 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (13893 ms) 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (2169 ms) 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (3479 ms) 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (14211 ms) 2023-12-01T17:09:13.099 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (1833 ms) 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4051 ms) 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (5861 ms) 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (52419 ms total) 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [----------] Global test environment tear-down 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (251020 ms total) 2023-12-01T17:09:13.100 INFO:tasks.workunit.client.0.smithi067.stdout: api_lock_pp: [ PASSED ] 16 tests. 2023-12-01T17:09:22.237 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:36.018 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.260 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:36.041 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.288 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:36.048 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.290 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:36.050 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.298 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:36.079 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.347 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:36.128 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.363 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:36.123 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.432 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:36.213 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:22.479 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:36.261 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:27.465 INFO:tasks.workunit.client.0.smithi067.stdout:sIoECPP.RoundTripPP2 (2792 ms) 2023-12-01T17:09:27.465 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2023-12-01T17:09:27.465 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (5793 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (1542 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (1006 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (3239 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (3220 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3322 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (215 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (9547 ms) 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2023-12-01T17:09:27.466 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (945 ms) 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (386 ms) 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2230 ms) 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (3224 ms) 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (67474 ms total) 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [----------] Global test environment tear-down 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (265457 ms total) 2023-12-01T17:09:27.467 INFO:tasks.workunit.client.0.smithi067.stdout: api_io_pp: [ PASSED ] 37 tests. 2023-12-01T17:09:42.167 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:55.948 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.170 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:55.951 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.234 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:55.993 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.237 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:55.997 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.256 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:56.037 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.260 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:56.041 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.296 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:06:56.077 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.312 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:06:56.072 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.343 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:06:56.124 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:09:42.410 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_1_[93020]: starting. 2023-12-01T17:09:42.410 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_1_[93020]: creating pool ceph_test_rados_list_parallel.smithi067-92951 2023-12-01T17:09:42.410 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_1_[93020]: created object 0... 2023-12-01T17:09:42.410 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_1_[93020]: created object 25... 2023-12-01T17:09:42.410 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_1_[93020]: created object 49... 2023-12-01T17:09:42.410 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_1_[93020]: finishing. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_1_[93020]: shutting down. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_2_[93023]: starting. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_2_[93023]: listing objects. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_2_[93023]: listed object 0... 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_2_[93023]: listed object 25... 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_2_[93023]: saw 50 objects 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_2_[93023]: shutting down. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_3_[132668]: starting. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_3_[132668]: creating pool ceph_test_rados_list_parallel.smithi067-92951 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_3_[132668]: created object 0... 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_3_[132668]: created object 25... 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_3_[132668]: created object 49... 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_3_[132668]: finishing. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_3_[132668]: shutting down. 2023-12-01T17:09:42.411 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_4_[132669]: starting. 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_4_[132669]: listing objects. 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_4_[132669]: listed object 0... 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_4_[132669]: listed object 25... 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_4_[132669]: saw 44 objects 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_4_[132669]: shutting down. 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_5_[132671]: starting. 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_5_[132671]: removed 25 objects... 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_5_[132671]: removed half of the objects 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_5_[132671]: removed 50 objects... 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_5_[132671]: removed 50 objects 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_5_[132671]: shutting down. 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.412 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_6_[143920]: starting. 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_6_[143920]: creating pool ceph_test_rados_list_parallel.smithi067-92951 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_6_[143920]: created object 0... 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_6_[143920]: created object 25... 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_6_[143920]: created object 49... 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_6_[143920]: finishing. 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_6_[143920]: shutting down. 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_7_[143921]: starting. 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_7_[143921]: listing objects. 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_7_[143921]: listed object 0... 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_7_[143921]: listed object 25... 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_7_[143921]: listed object 50... 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_7_[143921]: saw 57 objects 2023-12-01T17:09:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_7_[143921]: shutting down. 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_8_[143922]: starting. 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_8_[143922]: added 25 objects... 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_8_[143922]: added half of the objects 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_8_[143922]: added 50 objects... 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_8_[143922]: added 50 objects 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_8_[143922]: shutting down. 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:09:46.095 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: Running main() from gmock_main.cc 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] Global test environment set-up. 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (42945 ms) 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (5939 ms) 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (13575 ms) 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (5047 ms) 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (67506 ms total) 2023-12-01T17:09:46.096 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (26558 ms) 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (21125 ms) 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (50449 ms) 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (98132 ms total) 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (13186 ms) 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (3155 ms) 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2023-12-01T17:09:46.097 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (9741 ms) 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (3113 ms) 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (29195 ms total) 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (23893 ms) 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (16481 ms) 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (40374 ms total) 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [----------] Global test environment tear-down 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (283948 ms total) 2023-12-01T17:09:46.098 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots: [ PASSED ] 13 tests. 2023-12-01T17:09:52.593 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: Running main() from gmock_main.cc 2023-12-01T17:09:52.593 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2023-12-01T17:09:52.593 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [----------] Global test environment set-up. 2023-12-01T17:09:52.593 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [----------] 14 tests from LibRadosIo 2023-12-01T17:09:52.593 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2023-12-01T17:09:52.593 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (24786 ms) 2023-12-01T17:09:52.593 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: no timeout :/ 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: no timeout :/ 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: no timeout :/ 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (8241 ms) 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (76743 ms) 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.Checksum (4952 ms) 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (7882 ms) 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (1570 ms) 2023-12-01T17:09:52.594 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (10326 ms) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (5053 ms) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.TruncTest (104 ms) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (758 ms) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3699 ms) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.RmXattr (3073 ms) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIo.XattrIter (352 ms) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [----------] 14 tests from LibRadosIo (147539 ms total) 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: 2023-12-01T17:09:52.595 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (17515 ms) 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (4138 ms) 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (9088 ms) 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (3123 ms) 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (744 ms) 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (2338 ms) 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (6635 ms) 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2023-12-01T17:09:52.596 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (5428 ms) 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (30852 ms) 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (5359 ms) 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [----------] 10 tests from LibRadosIoEC (85221 ms total) 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [----------] Global test environment tear-down 2023-12-01T17:09:52.597 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [==========] 24 tests from 2 test suites ran. (290598 ms total) 2023-12-01T17:09:52.598 INFO:tasks.workunit.client.0.smithi067.stdout: api_io: [ PASSED ] 24 tests. 2023-12-01T17:09:56.869 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_1_[93154]: starting. 2023-12-01T17:09:56.869 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_1_[93154]: creating pool ceph_test_rados_delete_pools_parallel.smithi067-93085 2023-12-01T17:09:56.869 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_1_[93154]: created object 0... 2023-12-01T17:09:56.869 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_1_[93154]: created object 25... 2023-12-01T17:09:56.869 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_1_[93154]: created object 49... 2023-12-01T17:09:56.869 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_1_[93154]: finishing. 2023-12-01T17:09:56.869 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_1_[93154]: shutting down. 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_2_[93155]: starting. 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_2_[93155]: deleting pool ceph_test_rados_delete_pools_parallel.smithi067-93085 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_2_[93155]: shutting down. 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: ******************************* 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_3_[143061]: starting. 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_3_[143061]: creating pool ceph_test_rados_delete_pools_parallel.smithi067-93085 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_3_[143061]: created object 0... 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_3_[143061]: created object 25... 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_3_[143061]: created object 49... 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_3_[143061]: finishing. 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_3_[143061]: shutting down. 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: ******************************* 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_5_[143063]: starting. 2023-12-01T17:09:56.870 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_5_[143063]: listing objects. 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_5_[143063]: listed object 0... 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_5_[143063]: listed object 25... 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_5_[143063]: saw 50 objects 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_5_[143063]: shutting down. 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: ******************************* 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_4_[143062]: starting. 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_4_[143062]: deleting pool ceph_test_rados_delete_pools_parallel.smithi067-93085 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: process_4_[143062]: shutting down. 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: ******************************* 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: ******************************* 2023-12-01T17:09:56.871 INFO:tasks.workunit.client.0.smithi067.stdout: delete_pools_parallel: ******* SUCCESS ********** 2023-12-01T17:10:02.394 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:16.174 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.397 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:16.178 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.426 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:16.207 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.456 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:16.215 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.464 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:16.224 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.473 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:16.255 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.528 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:16.288 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.570 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:16.351 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:02.621 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:16.402 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:03.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2023-12-01T17:10:03.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [==========] Running 19 tests from 4 test suites. 2023-12-01T17:10:03.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2023-12-01T17:10:03.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: seed 92641 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (42019 ms) 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (43600 ms) 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (14315 ms) 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (6232 ms) 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (7507 ms) 2023-12-01T17:10:03.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (113673 ms total) 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (52416 ms) 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (28921 ms) 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (35103 ms) 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (8739 ms) 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (6246 ms) 2023-12-01T17:10:03.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.0.0-7570-gd710c09d/rpm/el9/BUILD/ceph-18.0.0-7570-gd710c09d/src/test/librados/snapshots_cxx.cc:463: Skipped 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi067-92641-7 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: waiting for snaps to purge 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (19867 ms) 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (151292 ms total) 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (5558 ms) 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2023-12-01T17:10:03.334 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (4617 ms) 2023-12-01T17:10:03.335 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2023-12-01T17:10:03.335 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (5276 ms) 2023-12-01T17:10:03.335 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2023-12-01T17:10:03.335 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2079 ms) 2023-12-01T17:10:03.335 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (17530 ms total) 2023-12-01T17:10:03.335 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: 2023-12-01T17:10:03.335 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN io: Running main() from gmock_main.cc 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [==========] Running 16 tests from 2 test suites. 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [----------] Global test environment set-up. 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [----------] 14 tests from NeoRadosIo 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.Limits 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.Limits (109320 ms) 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (34681 ms) 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.ReadOp (38975 ms) 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2023-12-01T17:10:06.479 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.SparseRead (20788 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.RoundTrip (9904 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (4930 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (5276 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (8706 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (6342 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.Trunc 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.Trunc (7665 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.Remove 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.Remove (8718 ms) 2023-12-01T17:10:06.480 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (9748 ms) 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.RmXattr (4633 ms) 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosIo.GetXattrs (4492 ms) 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [----------] 14 tests from NeoRadosIo (274178 ms total) 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [----------] 2 tests from NeoRadosECIo 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosECIo.SimpleWrite 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosECIo.SimpleWrite (17015 ms) 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ RUN ] NeoRadosECIo.ReadOp 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ OK ] NeoRadosECIo.ReadOp (12835 ms) 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [----------] 2 tests from NeoRadosECIo (29850 ms total) 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: 2023-12-01T17:10:06.481 INFO:tasks.workunit.client.0.smithi067.stdout: io: [----------] Global test environment tear-down 2023-12-01T17:10:06.482 INFO:tasks.workunit.client.0.smithi067.stdout: io: [==========] 16 tests from 2 test suites ran. (304029 ms total) 2023-12-01T17:10:06.482 INFO:tasks.workunit.client.0.smithi067.stdout: io: [ PASSED ] 16 tests. 2023-12-01T17:10:18.257 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: Running main() from gmock_main.cc 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [----------] Global test environment set-up. 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (19629 ms) 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (48521 ms) 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (48424 ms) 2023-12-01T17:10:18.258 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (37349 ms) 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (50473 ms) 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (204396 ms total) 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (19473 ms) 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (24846 ms) 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (24251 ms) 2023-12-01T17:10:18.259 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (14072 ms) 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (6249 ms) 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi067-93360-11. 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: Waiting for snaps to purge. 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (22458 ms) 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (111350 ms total) 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [----------] Global test environment tear-down 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (315746 ms total) 2023-12-01T17:10:18.260 INFO:tasks.workunit.client.0.smithi067.stdout: snapshots: [ PASSED ] 11 tests. 2023-12-01T17:10:22.562 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:36.343 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.568 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:36.349 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.635 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:36.394 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.638 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:36.399 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.644 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:36.424 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.649 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:36.430 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.697 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:36.478 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.699 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:36.459 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:22.735 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:36.516 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:33.938 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:47.719 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:33.948 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:47.729 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:34.021 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:47.802 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:34.023 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:47.803 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:34.032 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:47.792 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:34.038 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:47.798 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:34.041 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:47.821 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:34.092 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:47.852 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:34.128 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:47.909 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:39.365 INFO:tasks.workunit.client.0.smithi067.stdout: ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2023-12-01T17:10:39.365 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (19141 ms) 2023-12-01T17:10:39.365 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2023-12-01T17:10:39.365 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (7843 ms) 2023-12-01T17:10:39.366 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2023-12-01T17:10:39.366 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (5668 ms) 2023-12-01T17:10:39.366 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (32652 ms total) 2023-12-01T17:10:39.366 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: 2023-12-01T17:10:39.366 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2023-12-01T17:10:39.366 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [==========] 19 tests from 4 test suites ran. (337166 ms total) 2023-12-01T17:10:39.366 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ PASSED ] 18 tests. 2023-12-01T17:10:39.367 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2023-12-01T17:10:39.367 INFO:tasks.workunit.client.0.smithi067.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2023-12-01T17:10:40.279 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: Running main() from gmock_main.cc 2023-12-01T17:10:40.279 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2023-12-01T17:10:40.279 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [----------] Global test environment set-up. 2023-12-01T17:10:40.279 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [----------] 26 tests from LibRadosAio 2023-12-01T17:10:40.279 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2023-12-01T17:10:40.279 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.TooBig (5751 ms) 2023-12-01T17:10:40.279 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (83741 ms) 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (9539 ms) 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (33370 ms) 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (29060 ms) 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (41256 ms) 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (10184 ms) 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (12705 ms) 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2023-12-01T17:10:40.280 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4246 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (11940 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (6435 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (5259 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (7883 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (4133 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.Flush (3622 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (5382 ms) 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2023-12-01T17:10:40.281 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (6344 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (7818 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (6189 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (12048 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (7207 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4574 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (5745 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4972 ms) 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2023-12-01T17:10:40.282 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (4532 ms) 2023-12-01T17:10:40.283 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2023-12-01T17:10:40.283 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (4351 ms) 2023-12-01T17:10:40.283 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [----------] 26 tests from LibRadosAio (338287 ms total) 2023-12-01T17:10:40.283 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: 2023-12-01T17:10:40.285 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2023-12-01T17:10:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.Simp misc: Running main() from gmock_main.cc 2023-12-01T17:10:42.413 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [==========] Running 12 tests from 1 test suite. 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [----------] Global test environment set-up. 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [----------] 12 tests from NeoRadosMisc 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.Version 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.Version (5082 ms) 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (3116 ms) 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.LongName (274280 ms) 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (11885 ms) 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (12075 ms) 2023-12-01T17:10:42.414 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (8289 ms) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.Exec (3429 ms) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (5791 ms) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3896 ms) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.BigObject (5663 ms) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2284 ms) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (4142 ms) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [----------] 12 tests from NeoRadosMisc (339933 ms total) 2023-12-01T17:10:42.415 INFO:tasks.workunit.client.0.smithi067.stdout: misc: 2023-12-01T17:10:42.416 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [----------] Global test environment tear-down 2023-12-01T17:10:42.416 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [==========] 12 tests from 1 test suite ran. (339933 ms total) 2023-12-01T17:10:42.416 INFO:tasks.workunit.client.0.smithi067.stdout: misc: [ PASSED ] 12 tests. 2023-12-01T17:10:42.868 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:56.649 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:42.872 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:56.653 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:42.919 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:07:56.700 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:42.951 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:56.712 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:42.955 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:56.736 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:42.965 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:56.746 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:42.969 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:56.729 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:43.017 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:07:56.777 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:43.023 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:07:56.804 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: Running main() from gmock_main.cc 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] Global test environment set-up. 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (8677 ms) 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2023-12-01T17:10:50.700 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (98190 ms) 2023-12-01T17:10:50.701 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2023-12-01T17:10:50.701 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (25653 ms) 2023-12-01T17:10:50.701 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2023-12-01T17:10:50.701 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (28905 ms) 2023-12-01T17:10:50.701 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2023-12-01T17:10:50.701 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (41278 ms) 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (12064 ms) 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (8512 ms) 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (5546 ms) 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (10551 ms) 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (4531 ms) 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (8263 ms) 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2023-12-01T17:10:50.702 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (10098 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (12240 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (7390 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (5680 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (7925 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (6767 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3704 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (4126 ms) 2023-12-01T17:10:50.703 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (6161 ms) 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (4868 ms) 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (5792 ms) 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3772 ms) 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (6547 ms) 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3176 ms) 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3696 ms) 2023-12-01T17:10:50.704 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2023-12-01T17:10:55.229 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibR list_parallel: ******************************* 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_9_[153674]: starting. 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_9_[153674]: creating pool ceph_test_rados_list_parallel.smithi067-92951 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_9_[153674]: created object 0... 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_9_[153674]: created object 25... 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_9_[153674]: created object 49... 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_9_[153674]: finishing. 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_9_[153674]: shutting down. 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.230 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: starting. 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: listing objects. 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: listed object 0... 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: listed object 25... 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: listed object 50... 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: listed object 75... 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: saw 98 objects 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_10_[153675]: shutting down. 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_13_[153678]: starting. 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_13_[153678]: removed 25 objects... 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_13_[153678]: removed half of the objects 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_13_[153678]: removed 50 objects... 2023-12-01T17:10:55.231 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_13_[153678]: removed 50 objects 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_13_[153678]: shutting down. 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_11_[153676]: starting. 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_11_[153676]: added 25 objects... 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_11_[153676]: added half of the objects 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_11_[153676]: added 50 objects... 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_11_[153676]: added 50 objects 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_11_[153676]: shutting down. 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.232 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_12_[153677]: starting. 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_12_[153677]: added 25 objects... 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_12_[153677]: added half of the objects 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_12_[153677]: added 50 objects... 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_12_[153677]: added 50 objects 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_12_[153677]: shutting down. 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.233 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_14_[157978]: starting. 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_14_[157978]: creating pool ceph_test_rados_list_parallel.smithi067-92951 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_14_[157978]: created object 0... 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_14_[157978]: created object 25... 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_14_[157978]: created object 49... 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_14_[157978]: finishing. 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_14_[157978]: shutting down. 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: starting. 2023-12-01T17:10:55.234 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: listing objects. 2023-12-01T17:11:00.001 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: listed object 0... 2023-12-01T17:11:00.001 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: listed object 25... 2023-12-01T17:11:00.001 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: listed object 50... 2023-12-01T17:11:00.001 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: listed object 75... 2023-12-01T17:11:00.001 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: listed object 100... 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: listed object 125... 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: saw 150 objects 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_15_[157979]: shutting down. 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_16_[157980]: starting. 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_16_[157980]: added 25 objects... 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_16_[157980]: added half of the objects 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_16_[157980]: added 50 objects... 2023-12-01T17:11:00.002 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_16_[157980]: added 50 objects 2023-12-01T17:11:00.003 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: process_16_[157980]: shutting down. 2023-12-01T17:11:00.003 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.003 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.003 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.003 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.003 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******************************* 2023-12-01T17:11:00.003 INFO:tasks.workunit.client.0.smithi067.stdout: list_parallel: ******* SUCCESS ********** 2023-12-01T17:11:03.017 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:08:16.798 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.020 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:08:16.802 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.077 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:16.837 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.079 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:16.840 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.091 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:08:16.872 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.094 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:08:16.876 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.100 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:08:16.881 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.138 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:16.898 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:03.172 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:08:16.953 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.096 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:08:36.875 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.108 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:08:36.889 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.112 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:08:36.893 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.156 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:08:36.937 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.157 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:36.917 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.163 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:36.923 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.236 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:08:36.996 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.259 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:08:37.040 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:23.292 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:08:37.073 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:33.961 INFO:tasks.workunit.client.0.smithi067.stdout:list: : entry=9 expected=9 2023-12-01T17:11:33.961 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:de5d7c5f:::12:head 2023-12-01T17:11:33.961 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:de5d7c5f:::12:head expected=15:de5d7c5f:::12:head 2023-12-01T17:11:33.961 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:de5d7c5f:::12:head -> 12 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=12 expected=12 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:d83876eb:::4:head 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:d83876eb:::4:head expected=15:d83876eb:::4:head 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:d83876eb:::4:head -> 4 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=4 expected=4 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:cfc208b3:::3:head 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:cfc208b3:::3:head expected=15:cfc208b3:::3:head 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:cfc208b3:::3:head -> 3 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=3 expected=3 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:c4fdafeb:::6:head 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:c4fdafeb:::6:head expected=15:c4fdafeb:::6:head 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:c4fdafeb:::6:head -> 6 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=6 expected=6 2023-12-01T17:11:33.962 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:bd63b0f1:::8:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:bd63b0f1:::8:head expected=15:bd63b0f1:::8:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:bd63b0f1:::8:head -> 8 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=8 expected=8 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:b29083e3:::5:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:b29083e3:::5:head expected=15:b29083e3:::5:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:b29083e3:::5:head -> 5 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=5 expected=5 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:89d3ae78:::11:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:89d3ae78:::11:head expected=15:89d3ae78:::11:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:89d3ae78:::11:head -> 11 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=11 expected=11 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : seek to 15:863748b0:::15:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : cursor()=15:863748b0:::15:head expected=15:863748b0:::15:head 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: > 15:863748b0:::15:head -> 15 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: : entry=15 expected=15 2023-12-01T17:11:33.963 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (12059 ms) 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (172087 ms) 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: split 0/5 -> MIN 15:33333333::::head 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: split 1/5 -> 15:33333333::::head 15:66666666::::head 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: split 2/5 -> 15:66666666::::head 15:99999999::::head 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: split 3/5 -> 15:99999999::::head 15:cccccccc::::head 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: split 4/5 -> 15:cccccccc::::head MAX 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (29001 ms) 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] 7 tests from LibRadosList (373811 ms total) 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] 3 tests from LibRadosListEC 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (2644 ms) 2023-12-01T17:11:33.964 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset foo1,foo2,foo3 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo1 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo2 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo3 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset foo1,foo4,foo5 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo4 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo5 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo1 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset foo6,foo7 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo7 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: foo6 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2023-12-01T17:11:33.965 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns1:foo4 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns1:foo5 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns2:foo7 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns2:foo6 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: ns1:foo1 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: :foo1 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: :foo2 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: :foo3 2023-12-01T17:11:33.966 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (5034 ms) 2023-12-01T17:11:48.223 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:02.004 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.292 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:02.073 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.357 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:02.137 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.357 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:02.116 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.360 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:02.142 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.367 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:02.128 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.397 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:02.178 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.437 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:02.218 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:48.469 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:02.230 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:11:50.553 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 1 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 10 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 13 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 7 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 14 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 0 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 15 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 11 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 5 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 8 0 2023-12-01T17:11:50.554 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 6 0 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 3 0 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 4 0 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 12 0 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 9 0 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 2 0 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (8971 ms) 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] 3 tests from LibRadosListEC (16649 ms total) 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] 1 test from LibRadosListNP 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (4873 ms) 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] 1 test from LibRadosListNP (4873 ms total) 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: 2023-12-01T17:11:50.555 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [----------] Global test environment tear-down 2023-12-01T17:11:50.556 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [==========] 11 tests from 3 test suites ran. (408515 ms total) 2023-12-01T17:11:50.556 INFO:tasks.workunit.client.0.smithi067.stdout: api_list: [ PASSED ] 11 tests. 2023-12-01T17:12:08.346 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:22.127 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.352 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:22.133 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.412 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:22.193 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.416 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:22.197 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.418 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:22.178 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.419 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:22.180 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.423 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:22.204 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.475 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:22.256 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:08.476 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:22.237 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.381 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:38.161 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.435 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:38.216 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.482 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:38.263 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.511 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:38.292 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.513 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:38.294 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.518 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:38.278 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.519 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:38.279 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.561 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:38.342 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:24.562 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:38.323 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.217 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:41.998 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.279 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:42.060 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.309 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:09:42.089 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.340 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:42.100 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.343 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:42.125 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.344 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:42.105 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.346 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:42.128 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.368 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:09:42.149 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:28.368 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:09:42.128 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:37.532 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: Running main() from gmock_main.cc 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [----------] Global test environment set-up. 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (25600 ms) 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (90965 ms) 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (19154 ms) 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (37053 ms) 2023-12-01T17:12:37.533 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (27299 ms) 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (12243 ms) 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (242709 ms) 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (455023 ms total) 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [----------] Global test environment tear-down 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (455023 ms total) 2023-12-01T17:12:37.534 INFO:tasks.workunit.client.0.smithi067.stdout: write_operations: [ PASSED ] 7 tests. 2023-12-01T17:12:44.607 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: Running main() from gmock_main.cc 2023-12-01T17:12:44.607 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2023-12-01T17:12:44.607 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [----------] Global test environment set-up. 2023-12-01T17:12:44.607 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2023-12-01T17:12:44.607 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2023-12-01T17:12:44.607 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (12860 ms) 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (81676 ms) 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (8901 ms) 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (32998 ms) 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (36724 ms) 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2023-12-01T17:12:44.608 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (30736 ms) 2023-12-01T17:12:44.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2023-12-01T17:12:44.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (258434 ms) 2023-12-01T17:12:44.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (462329 ms total) 2023-12-01T17:12:44.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: 2023-12-01T17:12:44.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [----------] Global test environment tear-down 2023-12-01T17:12:44.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (462329 ms total) 2023-12-01T17:12:44.609 INFO:tasks.workunit.client.0.smithi067.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2023-12-01T17:12:48.463 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:02.244 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.467 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:02.249 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.517 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:02.278 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.520 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:02.280 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.525 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:02.306 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.528 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:02.310 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.567 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:02.348 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.574 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:02.334 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:48.580 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:02.361 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout:leWrite 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (8250 ms) 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (8043 ms) 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (9601 ms) 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7969 ms) 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7672 ms) 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2023-12-01T17:12:49.309 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7269 ms) 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (8367 ms) 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7238 ms) 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (8939 ms) 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7805 ms) 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (8144 ms) 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (8797 ms) 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2023-12-01T17:12:49.310 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (8045 ms) 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7102 ms) 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7299 ms) 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (8488 ms) 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (129029 ms total) 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [----------] Global test environment tear-down 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (467316 ms total) 2023-12-01T17:12:49.311 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio: [ PASSED ] 42 tests. 2023-12-01T17:13:08.557 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:22.338 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.562 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:22.343 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.613 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:22.394 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.616 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:22.376 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.617 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:22.378 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.630 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:22.411 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.633 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:22.415 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.661 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:22.421 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:08.679 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:22.461 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:10.026 INFO:tasks.workunit.client.0.smithi067.stdout: list: Running main() from gmock_main.cc 2023-12-01T17:13:10.026 INFO:tasks.workunit.client.0.smithi067.stdout: list: [==========] Running 6 tests from 2 test suites. 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [----------] Global test environment set-up. 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [----------] 3 tests from NeoradosList 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ RUN ] NeoradosList.ListObjects 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ OK ] NeoradosList.ListObjects (115461 ms) 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ OK ] NeoradosList.ListObjectsNS (57326 ms) 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ OK ] NeoradosList.ListObjectsMany (222195 ms) 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [----------] 3 tests from NeoradosList (394982 ms total) 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [----------] 3 tests from NeoradosECList 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ RUN ] NeoradosECList.ListObjects 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ OK ] NeoradosECList.ListObjects (8191 ms) 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ RUN ] NeoradosECList.ListObjectsNS 2023-12-01T17:13:10.027 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ OK ] NeoradosECList.ListObjectsNS (12462 ms) 2023-12-01T17:13:10.028 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ RUN ] NeoradosECList.ListObjectsMany 2023-12-01T17:13:10.028 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ OK ] NeoradosECList.ListObjectsMany (71931 ms) 2023-12-01T17:13:10.028 INFO:tasks.workunit.client.0.smithi067.stdout: list: [----------] 3 tests from NeoradosECList (92584 ms total) 2023-12-01T17:13:10.028 INFO:tasks.workunit.client.0.smithi067.stdout: list: 2023-12-01T17:13:10.028 INFO:tasks.workunit.client.0.smithi067.stdout: list: [----------] Global test environment tear-down 2023-12-01T17:13:10.028 INFO:tasks.workunit.client.0.smithi067.stdout: list: [==========] 6 tests from 2 test suites ran. (487566 ms total) 2023-12-01T17:13:10.028 INFO:tasks.workunit.client.0.smithi067.stdout: list: [ PASSED ] 6 tests. 2023-12-01T17:13:28.705 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:42.485 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.708 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:42.489 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.709 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:42.469 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.713 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:42.472 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.764 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:42.545 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.767 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:10:42.527 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.769 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:10:42.550 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.769 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:42.551 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:28.811 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:10:42.593 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:38.616 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2023-12-01T17:13:38.616 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] Global test environment set-up. 2023-12-01T17:13:38.616 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: seed 92416 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (348 ms) 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (98105 ms) 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (15231 ms) 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (21788 ms) 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2023-12-01T17:13:38.617 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (6093 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (1086 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (53 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (91 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2873 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (3468 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (107 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (3102 ms) 2023-12-01T17:13:38.618 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (2442 ms) 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: osd_max_attr_size = 0 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (268472 ms) 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (8341 ms) 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: waiting for initial deep scrubs... 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: done waiting, doing copies 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: waiting for final deep scrubs... 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: done waiting 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (74168 ms) 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (98 ms) 2023-12-01T17:13:38.619 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (404 ms) 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (5328 ms) 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2023-12-01T17:13:38.620 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2023-12-01T17:13:43.994 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----adosAio.RoundTripAppendPP (4595 ms) 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3565 ms) 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3601 ms) 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3540 ms) 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: setting pool EIO 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: waiting for 745 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: waiting for 745 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: waiting for 745 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: max_success 255, min_failed 256 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (16191 ms) 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4245 ms) 2023-12-01T17:13:43.995 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (379850 ms total) 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3301 ms) 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (10217 ms) 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (18264 ms) 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3296 ms) 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (35078 ms total) 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3466 ms) 2023-12-01T17:13:43.996 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3466 ms total) 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14734 ms) 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7579 ms) 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7086 ms) 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7757 ms) 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3257 ms) 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (8523 ms) 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2023-12-01T17:13:43.997 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (8237 ms) 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7140 ms) 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7367 ms) 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (6807 ms) 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (8541 ms) 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7970 ms) 2023-12-01T17:13:43.998 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2023-12-01T17:13:49.083 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:02.864 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.089 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:02.870 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.160 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:02.920 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.162 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:02.943 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.162 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:02.922 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.166 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:02.948 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.174 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:02.955 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.206 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:02.966 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:13:49.251 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:03.032 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio------] 22 tests from LibRadosMiscPP (511598 ms total) 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (2756 ms) 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (2756 ms total) 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: 2023-12-01T17:14:08.331 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (517 ms) 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (47 ms) 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (564 ms total) 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (468 ms) 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (53 ms) 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (521 ms total) 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: 2023-12-01T17:14:08.332 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (693 ms) 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (41 ms) 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (734 ms total) 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1604 ms) 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1604 ms total) 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [----------] Global test environment tear-down 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (546230 ms total) 2023-12-01T17:14:08.333 INFO:tasks.workunit.client.0.smithi067.stdout: api_misc_pp: [ PASSED ] 31 tests. 2023-12-01T17:14:09.215 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:22.996 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.233 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:23.011 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.265 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:23.025 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.266 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:23.027 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.309 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:23.090 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.311 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:23.092 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.321 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:23.081 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.350 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:23.131 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:09.383 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:23.165 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:10.354 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:24.114 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:10.355 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:24.115 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:10.377 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:24.137 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:10.790 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:24.572 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:10.792 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:24.573 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:10.817 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:24.598 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:11.019 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:24.801 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:11.020 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:24.802 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:11.036 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:24.817 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:31.249 INFO:tasks.workunit.client.0.smithi067.stdout:_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (8611 ms) 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (4095 ms) 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6967 ms) 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7128 ms) 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6967 ms) 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2023-12-01T17:14:31.250 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7283 ms) 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7064 ms) 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7748 ms) 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (150862 ms total) 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [----------] Global test environment tear-down 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (569256 ms total) 2023-12-01T17:14:31.251 INFO:tasks.workunit.client.0.smithi067.stdout: api_aio_pp: [ PASSED ] 57 tests. 2023-12-01T17:14:34.189 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:47.970 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.193 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:47.975 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.241 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:48.001 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.243 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:48.004 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.244 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:48.026 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.246 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:11:48.028 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.248 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:48.029 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.289 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:11:48.050 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:34.294 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:11:48.075 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.454 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:08.235 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.456 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:08.237 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.500 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:08.260 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.501 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:08.261 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.502 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:08.284 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.511 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:08.292 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.518 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:08.300 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.538 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:08.298 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:14:54.566 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:08.347 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.462 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:23.244 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.466 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:23.247 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.496 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:23.278 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.510 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:23.270 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.514 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:23.272 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.516 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:23.297 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.519 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:23.300 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.540 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:23.300 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:09.557 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:23.338 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.507 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:28.288 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.567 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:28.348 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.581 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:28.362 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.610 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:28.371 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.611 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:28.372 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.623 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:28.405 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.624 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:28.384 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.629 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:28.410 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:14.647 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:28.428 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.409 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:38.190 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.415 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:38.196 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.441 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:38.222 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.463 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:38.223 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.466 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:38.247 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.468 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:38.228 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.469 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:38.251 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.486 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:38.245 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:24.490 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:38.272 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.441 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:48.222 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.444 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:48.226 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.459 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:48.241 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.496 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:48.256 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.497 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:48.278 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.500 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:48.281 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.504 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:48.264 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.513 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:48.295 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:34.517 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:48.278 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.468 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:58.249 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.469 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:58.251 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.483 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:12:58.264 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.516 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:58.297 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.518 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:58.300 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.528 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:58.288 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.529 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:58.290 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.531 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:12:58.312 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:44.541 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:12:58.301 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.648 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:08.429 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.717 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:08.499 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.746 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:08.527 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.781 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:08.541 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.788 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:08.548 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.790 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:08.572 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.794 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:08.576 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.810 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:08.570 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:54.826 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:08.607 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.629 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:13.411 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.687 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:13.468 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.705 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:13.487 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.734 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:13.494 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.736 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:13.496 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.751 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:13.533 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.753 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:13.513 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.753 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:13.534 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:15:59.778 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:13.559 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.627 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:28.408 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.628 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:28.410 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.630 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:28.412 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.661 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:28.442 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.679 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:28.460 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.711 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:28.493 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.721 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:28.480 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.722 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:28.482 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:14.747 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:28.507 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.596 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:29.356 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.597 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:29.357 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.607 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:29.367 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.724 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:29.505 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.725 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:29.507 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.726 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:29.508 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.727 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:29.509 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.741 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:29.522 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:15.742 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:29.523 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.765 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:43.546 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.773 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:43.555 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.809 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:43.591 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.814 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:43.575 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.817 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:43.577 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.823 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:43.604 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.835 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:43.616 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.848 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:43.608 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:29.879 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:43.660 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.737 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:48.518 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.742 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:48.523 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.759 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:48.540 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.801 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:48.561 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.801 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:48.579 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.801 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:48.582 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.804 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:48.564 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.819 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:48.600 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:34.834 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:48.594 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.897 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:58.679 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.902 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:58.684 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.927 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:13:58.708 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.950 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:58.710 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.954 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:58.714 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.956 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:58.738 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.957 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:58.739 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.978 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:13:58.739 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:44.983 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:13:58.765 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] Global test environment set-up. 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: seed 92454 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (24954 ms) 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (140431 ms) 2023-12-01T17:16:53.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (1157 ms) 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (166542 ms total) 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (44317 ms) 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (57444 ms) 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (42015 ms) 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: my_snaps [4] 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: my_snaps [5,4] 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: my_snaps [6,5,4] 2023-12-01T17:16:53.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: my_snaps [7,6,5,4] 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: promoting some heads 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: promoting from clones for snap 7 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: promoting from clones for snap 6 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: promoting from clones for snap 5 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: promoting from clones for snap 4 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: waiting for scrubs... 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: done waiting 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (187683 ms) 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (12140 ms) 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (10240 ms) 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8507 ms) 2023-12-01T17:16:53.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (9943 ms) 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (12143 ms) 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (10855 ms) 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (11426 ms) 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (41892 ms) 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (10140 ms) 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (12111 ms) 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2023-12-01T17:16:53.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (16492 ms) 2023-12-01T17:16:53.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2023-12-01T17:16:53.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (9149 ms) 2023-12-01T17:16:53.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2023-12-01T17:16:53.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (12052 ms) 2023-12-01T17:16:59.812 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:13.593 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.815 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:13.597 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.861 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:13.621 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.863 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:13.624 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.868 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:13.650 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.882 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:13.664 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.887 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:13.668 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.891 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:13.652 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:16:59.919 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:13.699 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.701 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:14.461 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.702 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:14.462 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.713 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:14.474 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.816 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:14.597 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.817 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:14.598 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.826 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:14.607 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.852 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:14.633 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.853 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:14.634 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:00.862 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:14.643 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:16.294 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2023-12-01T17:17:16.294 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-01T17:17:16.294 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: ok, hit_set contains 266:602f83fe:::foo:head 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (10245 ms) 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg_num = 32 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 0 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 1 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 2 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 3 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 4 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 5 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 6 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 7 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 8 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 9 ls 1701451031,0 2023-12-01T17:17:16.295 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 10 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 11 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 12 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 13 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 14 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 15 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 16 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 17 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 18 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 19 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 20 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 21 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 22 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 23 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 24 ls 1701451031,0 2023-12-01T17:17:16.296 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 25 ls 1701451031,0 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 26 ls 1701451031,0 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 27 ls 1701451031,0 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 28 ls 1701451031,0 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 29 ls 1701451031,0 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 30 ls 1701451031,0 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg 31 ls 1701451031,0 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: pg_num = 32 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6cac518f:::0:head 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:02547ec2:::1:head 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f905c69b:::2:head 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:cfc208b3:::3:head 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d83876eb:::4:head 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b29083e3:::5:head 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c4fdafeb:::6:head 2023-12-01T17:17:16.297 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:5c6b0b28:::7:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:bd63b0f1:::8:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:e960b815:::9:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:52ea6a34:::10:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:89d3ae78:::11:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:de5d7c5f:::12:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:566253c9:::13:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:62a1935d:::14:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:863748b0:::15:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:3958e169:::16:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:4d4dabf9:::17:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:8391935d:::18:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:28883081:::19:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:69259c59:::20:head 2023-12-01T17:17:16.298 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:4bdb80b7:::21:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a11c5d71:::22:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:271af37b:::23:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:95b121be:::24:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:58d1031b:::25:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:0a050783:::26:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c709704c:::27:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:cbe56eaf:::28:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:86b4b162:::29:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:70d89383:::30:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:dd450c7c:::31:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6d5729b1:::32:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c388f3fb:::33:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:56cfea31:::34:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:9dbc1bf7:::35:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:40b74ccd:::36:head 2023-12-01T17:17:16.299 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:4d5aaf42:::37:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:920f362c:::38:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6cc53222:::39:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:9cad833f:::40:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:1ea84d41:::41:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c4480ef6:::42:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a694361e:::43:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d1bd33e9:::44:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:ddc2cd5d:::45:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:2b782207:::46:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:7b187fca:::47:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:90ecdf6f:::48:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a5ed95fe:::49:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:ea0eaa55:::50:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f33ef17b:::51:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a0d1b2f6:::52:head 2023-12-01T17:17:16.300 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:60c5229e:::53:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:edcbc575:::54:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:102cf253:::55:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:efb7fb0b:::56:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:50d0a326:::57:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d4dc5daf:::58:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:3a130462:::59:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:ec87ed71:::60:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d5bc9454:::61:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:3ddfe313:::62:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:7c2816b9:::63:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:47e00e4d:::64:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c6410c18:::65:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b48ed237:::66:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:cd63ad31:::67:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b179e92b:::68:head 2023-12-01T17:17:16.301 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:0d9f741a:::69:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6d3352ae:::70:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c6d5c19e:::71:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:bc4729c3:::72:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:77e930b9:::73:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:0abeecfd:::74:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b7c37e15:::75:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b6378398:::76:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:02bd68de:::77:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:cc795d2d:::78:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:630d4fea:::79:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:e0d29ef5:::80:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:fd6f13d2:::81:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:606461d5:::82:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:eadbdc43:::83:head 2023-12-01T17:17:16.302 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:8761d0bb:::84:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:9ef0186f:::85:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:e0d41294:::86:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:961de695:::87:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:1423148f:::88:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:633a8fa2:::89:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a8653809:::90:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:3dac8b33:::91:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:35aad435:::92:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f6dcc343:::93:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:dbbdad87:::94:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:1cb48ce0:::95:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:03cd461c:::96:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:17a4ea99:::97:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:9993c9a7:::98:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6394211c:::99:head 2023-12-01T17:17:16.303 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:94c7ae57:::100:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6fdee5bb:::101:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:9a477fd1:::102:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:eb850916:::103:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:affc56b9:::104:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b42dc814:::105:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f319f8f0:::106:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:9a40b9de:::107:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:8b524f28:::108:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:e3de589f:::109:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:90f90a5b:::110:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a7b4f1d7:::111:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:af51766e:::112:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b6f90bd1:::113:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:e0261208:::114:head 2023-12-01T17:17:16.304 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c9569ef7:::115:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:61bebe50:::116:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:fe93412b:::117:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d3d38bee:::118:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:3100ba0c:::119:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d0560ada:::120:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f0ea8b35:::121:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:766f231a:::122:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a07a2582:::123:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:bd7c6b3a:::124:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:fb2ddaff:::125:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:4408e1fe:::126:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:ee1df7a7:::127:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c3002909:::128:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:4f48ffa9:::129:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:edf38733:::130:head 2023-12-01T17:17:16.305 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c08425c0:::131:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:5f902d98:::132:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:41ea2c93:::133:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:813cee13:::134:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:0131818d:::135:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:26ba5a85:::136:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:381b8a5a:::137:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:28797e47:::138:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:bfca7f22:::139:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:36807075:::140:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:80b03975:::141:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:5c15709b:::142:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f39ea15e:::143:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:ea992956:::144:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:48887b1c:::145:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:9f24a9dd:::146:head 2023-12-01T17:17:16.306 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:987f100b:::147:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d2dd3581:::148:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:7fed1808:::149:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c80b70e9:::150:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:85ed90f9:::151:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:36428b24:::152:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d044c34a:::153:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:7c18bf58:::154:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d1c21232:::155:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a7a3c575:::156:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:87da0633:::157:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d5ac3822:::158:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:3f20522d:::159:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6ca26563:::160:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:532ce135:::161:head 2023-12-01T17:17:16.307 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:c78863e6:::162:head 2023-12-01T17:17:20.078 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:33.859 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.080 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:33.862 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.127 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:33.909 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.132 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:33.892 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.133 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:33.893 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.146 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:33.928 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.149 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:33.931 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.164 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:33.924 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:20.180 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:33.962 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:29.950 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:43.731 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:29.952 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:43.733 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:29.980 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:43.760 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:29.995 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:43.755 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:29.997 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:43.758 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:30.009 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:43.791 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:30.014 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:43.796 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:30.015 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:43.775 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:30.039 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:43.820 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:39.977 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:53.758 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:39.979 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:53.761 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:40.000 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:14:53.782 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:40.025 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:53.786 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:40.026 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:53.808 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:40.027 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:53.788 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:40.030 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:53.811 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:40.046 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:14:53.806 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:40.051 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:14:53.832 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.190 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:03.971 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.192 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:03.974 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.215 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:03.975 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.217 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:03.978 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.221 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:04.003 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.239 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:03.999 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.271 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:04.053 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.276 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:04.057 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:17:50.301 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:04.083 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.056 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:13.837 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.057 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:13.839 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.089 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:13.870 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.098 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:13.858 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.100 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:13.861 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.106 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:13.887 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.109 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:13.891 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.123 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:13.884 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:00.137 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:13.919 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.201 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:18.982 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.206 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:18.987 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.223 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:19.004 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.262 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:19.043 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.264 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:19.046 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.266 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:19.026 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.267 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:19.028 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.281 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:19.062 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:05.285 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:19.046 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.161 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:33.942 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.164 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:33.945 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.187 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:33.968 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.208 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:33.969 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.209 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:33.970 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.211 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:33.993 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.213 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:33.995 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.231 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:33.991 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.241 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:34.022 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.962 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:34.723 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.963 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:34.724 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.972 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:34.733 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.989 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:34.771 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:20.990 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:34.772 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:21.002 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:34.784 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:21.006 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:34.788 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:21.007 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:34.789 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:21.015 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:34.797 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:26.938 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:7462ddf6:::163:head 2023-12-01T17:18:26.938 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:5d165639:::164:head 2023-12-01T17:18:26.938 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f43765fc:::165:head 2023-12-01T17:18:26.938 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b4c720e9:::166:head 2023-12-01T17:18:26.938 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:e694b040:::167:head 2023-12-01T17:18:26.938 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:afa38db2:::168:head 2023-12-01T17:18:26.938 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:77ba9f53:::169:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:87495034:::170:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:7c96bf0e:::171:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:dbe346cc:::172:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:e943ec24:::173:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f97a9c0c:::174:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:6f26e74d:::175:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:4f95e106:::176:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:0e6f2f8f:::177:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:05db05f1:::178:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:38a78d66:::179:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:d095610b:::180:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:a1a9d709:::181:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:1e5d39db:::182:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:f7df4fb9:::183:head 2023-12-01T17:18:26.939 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:03a7f161:::184:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:ba70721e:::185:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:28e5662d:::186:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:973d52de:::187:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:4303eb1c:::188:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:b990b48e:::189:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:29b8165b:::190:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:3547f197:::191:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:7e260936:::192:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:1abec7b1:::193:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:10fdda93:::194:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:15817eea:::195:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:770bab57:::196:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:ed9e13e7:::197:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:71471a8f:::198:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: checking for 267:10fb1d02:::199:head 2023-12-01T17:18:26.940 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (22479 ms) 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: first is 1701451055 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,1701451060,1701451061,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,1701451060,1701451061,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,1701451060,1701451061,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,1701451060,1701451061,1701451063,1701451064,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,1701451060,1701451061,1701451063,1701451064,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451055,1701451057,1701451058,1701451060,1701451061,1701451063,1701451064,0 2023-12-01T17:18:26.941 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451058,1701451060,1701451061,1701451063,1701451064,1701451066,1701451067,0 2023-12-01T17:18:26.942 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: first now 1701451058, trimmed 2023-12-01T17:18:26.942 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22452 ms) 2023-12-01T17:18:26.942 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2023-12-01T17:18:26.942 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: foo0 2023-12-01T17:18:26.942 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-01T17:18:26.942 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (16530 ms) 2023-12-01T17:18:26.942 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2023-12-01T17:18:28.187 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:41.968 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.194 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:41.975 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.230 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:42.011 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.262 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:42.022 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.268 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:42.049 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.269 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:42.029 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.282 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:42.063 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.291 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:42.052 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:28.313 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:42.094 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.295 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:49.077 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.299 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:49.081 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.327 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:49.108 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.346 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:49.127 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.347 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:49.108 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.350 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:49.111 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.351 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:49.132 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.372 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:49.132 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:35.372 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:49.154 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.304 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:54.086 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.308 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:54.090 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.329 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:15:54.110 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.353 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:54.113 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.355 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:54.116 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.372 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:15:54.132 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.379 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:54.161 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.382 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:54.164 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:40.401 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:15:54.182 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:46.778 INFO:tasks.workunit.client.0.smithi067.stdout: a api_watch_notify_pp: Running main() from gmock_main.cc 2023-12-01T17:18:46.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2023-12-01T17:18:46.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2023-12-01T17:18:46.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2023-12-01T17:18:46.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2023-12-01T17:18:46.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: notify 2023-12-01T17:18:46.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (804194 ms) 2023-12-01T17:18:46.779 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2023-12-01T17:18:46.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (249 ms) 2023-12-01T17:18:46.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (804443 ms total) 2023-12-01T17:18:46.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: 2023-12-01T17:18:46.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2023-12-01T17:18:46.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2023-12-01T17:18:46.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: notify 2023-12-01T17:18:46.780 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (498 ms) 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: notify 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4529 ms) 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (204 ms) 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (177 ms) 2023-12-01T17:18:46.781 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873218 notifier_gid 23050 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (125 ms) 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873219 notifier_gid 23050 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (143 ms) 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873219 notifier_gid 23050 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (126 ms) 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873220 notifier_gid 23050 2023-12-01T17:18:46.782 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (133 ms) 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873220 notifier_gid 23050 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (121 ms) 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873221 notifier_gid 23050 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (128 ms) 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: trying... 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873217 notifier_gid 23050 2023-12-01T17:18:46.783 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: timed out 2023-12-01T17:18:50.491 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:04.273 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.496 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:04.277 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.516 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:04.298 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.545 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:04.327 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.551 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:04.332 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.551 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:04.312 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.554 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:04.314 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.572 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:04.353 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:18:50.578 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:04.339 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:02.034 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: flushing 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: flushed 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3094 ms) 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: trying... 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2774548873218 notifier_gid 23050 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: timed out 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: flushing 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: flushed 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3106 ms) 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: List watches 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: notify2 2023-12-01T17:19:02.035 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2787433775107 notifier_gid 23050 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: notify2 done 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: watch_check 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: unwatch2 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: flushing 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: done 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3248 ms) 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: List watches 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: notify2 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: handle_notify cookie 94151157337008 notify_id 2796023709701 notifier_gid 23050 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: notify2 done 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: watch_check 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: unwatch2 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: flushing 2023-12-01T17:19:02.036 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: done 2023-12-01T17:19:02.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3139 ms) 2023-12-01T17:19:02.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18772 ms total) 2023-12-01T17:19:02.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: 2023-12-01T17:19:02.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2023-12-01T17:19:02.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (839792 ms total) 2023-12-01T17:19:02.037 INFO:tasks.workunit.client.0.smithi067.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2023-12-01T17:19:10.491 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:24.272 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.494 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:24.275 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.538 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:24.297 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.538 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:24.318 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.539 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:24.300 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.553 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:24.334 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.556 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:24.337 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.581 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:24.341 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:10.597 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:24.379 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.626 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:25.386 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.627 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:25.387 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.636 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:25.396 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.829 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:25.610 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.830 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:25.611 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.844 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:25.625 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.955 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:25.737 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.957 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:25.738 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:11.968 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:25.749 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.549 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:39.330 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.551 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:39.333 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.583 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:39.365 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.605 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:39.386 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.606 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:39.366 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.608 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:39.368 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.610 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:39.392 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.635 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:39.396 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:25.646 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:39.427 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.562 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:44.343 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.566 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:44.348 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.580 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:44.362 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.612 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:44.372 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.613 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:44.374 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.618 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:44.400 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.621 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:44.403 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.624 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:44.385 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:30.634 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:44.415 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.723 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:59.505 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.726 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:59.507 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.744 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:16:59.526 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.781 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:59.560 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.781 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:59.562 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.781 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:59.539 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.781 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:59.540 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.796 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:16:59.557 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:45.805 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:16:59.586 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.636 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:04.396 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.637 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:04.398 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.639 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:04.420 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.643 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:04.424 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.649 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:04.410 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.659 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:04.440 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.693 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:04.474 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.694 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:04.476 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:19:50.709 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:04.491 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.703 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:19.484 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.705 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:19.487 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.731 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:19.512 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.753 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:19.513 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.755 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:19.516 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.758 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:19.540 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.767 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:19.548 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.778 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:19.537 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:05.791 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:19.572 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.747 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:24.528 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.798 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:24.580 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.811 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:24.593 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.841 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:24.601 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.845 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:24.604 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.846 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:24.627 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.849 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:24.630 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.856 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:24.616 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:10.862 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:24.644 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:20.929 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:34.710 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:20.997 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:34.779 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:21.025 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:34.806 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:21.061 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:34.821 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:21.062 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:34.823 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:21.081 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:34.841 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:21.160 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:34.941 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:21.166 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:34.947 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:21.196 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:34.978 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:35.936 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:49.717 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:35.937 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:49.719 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:35.965 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:49.746 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:35.981 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:49.762 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:35.986 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:49.743 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:35.986 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:49.745 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:35.986 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:49.768 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.011 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:49.771 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.015 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:49.796 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.755 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:50.514 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.756 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:50.515 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.762 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:50.522 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.809 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:50.591 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.810 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:50.592 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.812 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:50.594 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.814 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:50.596 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.820 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:50.601 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:36.823 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:50.605 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.270 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:54.052 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.275 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:54.056 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.295 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:17:54.076 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.330 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:54.091 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.332 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:54.114 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.334 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:54.116 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.336 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:54.097 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.348 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:17:54.130 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:40.349 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:17:54.109 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.155 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:09.936 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.164 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:09.946 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.191 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:09.972 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.229 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:09.989 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.234 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:09.994 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.242 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:10.024 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.252 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:10.013 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.253 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:10.034 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:20:56.287 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:10.068 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.101 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:14.882 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.103 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:14.885 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.118 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:14.899 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.144 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:14.905 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.148 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:14.908 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.152 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:14.934 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.156 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:14.938 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.162 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:14.922 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:01.178 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:14.959 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.238 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:30.019 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.246 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:30.028 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.272 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:30.053 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.293 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:30.053 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.294 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:30.055 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.307 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:30.088 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.308 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:30.090 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.315 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:30.076 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:16.330 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:30.111 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.085 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:39.866 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.088 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:39.870 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.110 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:39.892 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.139 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:39.899 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.141 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:39.923 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.142 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:39.903 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.160 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:39.942 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.162 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:39.922 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:26.181 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:39.963 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.124 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:49.906 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.126 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:49.908 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.144 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:49.926 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.173 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:49.934 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.174 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:49.955 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.174 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:49.935 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.177 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:49.958 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.191 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:49.952 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:36.196 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:49.978 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.087 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:54.868 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.136 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:54.918 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.153 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:18:54.934 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.184 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:54.944 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.186 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:54.946 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.196 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:18:54.957 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.208 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:54.989 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.209 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:54.991 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:41.221 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:18:55.002 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:50.008 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:21:49.990+0000 24a81640 -1 snap_mapper.add_oid found existing snaps mapped on 287:336c6209:test-rados-api-smithi067-92454-80::foo:2, removing 2023-12-01T17:21:50.012 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:21:50.006+0000 2de93640 -1 snap_mapper.add_oid found existing snaps mapped on 287:336c6209:test-rados-api-smithi067-92454-80::foo:2, removing 2023-12-01T17:21:56.349 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:10.110 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.350 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:10.111 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.351 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:10.133 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.356 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:10.138 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.380 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:10.140 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.392 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:10.172 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.428 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:10.209 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.437 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:10.218 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:21:56.467 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:10.248 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.300 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:15.082 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.303 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:15.085 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.324 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:15.104 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.365 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:15.126 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.368 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:15.128 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.368 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:15.150 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.376 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:15.157 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.387 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:15.147 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:01.396 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:15.178 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.372 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:30.153 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.435 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:30.196 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.437 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:30.197 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.447 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:30.229 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.470 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:30.230 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.496 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:30.277 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.539 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:30.320 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.543 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:30.325 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:16.590 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:30.371 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.353 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:31.114 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.354 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:31.115 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.365 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:31.126 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.706 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:31.487 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.775 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:31.557 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.791 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:31.573 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.812 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:31.594 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.870 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:31.651 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:17.880 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:31.661 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout:pi_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (21936 ms) 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (26558 ms) 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4404 ms) 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (6369 ms) 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2023-12-01T17:22:27.674 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4657 ms) 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (5608 ms) 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (5517 ms) 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40655 ms) 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (18466 ms) 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4299 ms) 2023-12-01T17:22:27.675 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (4572 ms) 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (27216 ms) 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16039 ms) 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (7613 ms) 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5556 ms) 2023-12-01T17:22:27.676 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: waiting for scrubs... 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: done waiting 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (26939 ms) 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (11986 ms) 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (10613 ms) 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (11647 ms) 2023-12-01T17:22:27.677 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2023-12-01T17:22:31.407 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:45.188 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.410 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:45.191 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.453 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:45.234 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.455 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:45.216 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.458 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:45.218 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.471 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:45.252 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.474 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:45.256 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.485 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:45.245 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:31.511 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:45.292 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.556 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:50.337 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.559 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:50.340 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.574 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:19:50.355 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.608 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:50.390 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.611 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:50.392 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.625 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:19:50.406 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.633 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:50.394 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.644 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:50.405 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:36.658 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:19:50.419 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.665 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:00.447 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.674 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:00.455 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.709 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:00.491 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.734 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:00.495 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.737 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:00.498 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.752 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:00.534 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.767 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:00.528 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.843 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:00.624 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:46.896 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:00.678 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.038 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:08.820 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.043 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:08.824 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.070 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:08.852 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.101 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:08.861 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.118 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:08.900 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.119 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:08.880 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.122 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:08.903 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.148 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:08.909 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:22:55.155 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:08.936 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.536 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:15.318 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.538 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:15.320 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.552 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:15.334 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.586 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:15.346 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.588 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:15.349 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.601 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:15.361 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.605 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:15.386 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.610 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:15.390 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:01.620 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:15.402 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.768 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:20.550 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.771 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:20.552 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.783 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:20.564 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.816 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:20.576 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.816 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:20.577 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.828 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:20.588 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.828 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:20.610 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.845 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:20.625 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:06.860 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:20.642 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.649 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:35.430 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.654 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:35.435 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.690 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:35.471 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.697 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:35.479 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.700 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:35.461 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.701 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:35.462 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.702 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:35.484 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.726 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:35.484 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:21.726 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:35.507 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.464 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:36.224 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.465 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:36.225 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.472 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:36.233 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.542 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:36.322 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.542 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:36.324 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.545 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:36.327 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.547 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:36.328 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.551 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:36.332 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:22.555 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:36.337 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:31.742 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:23:31.735+0000 2be8f640 -1 snap_mapper.add_oid found existing snaps mapped on 106:076ecd41:test-rados-api-smithi067-92454-97::foo:31, removing 2023-12-01T17:23:31.761 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:23:31.717+0000 24680640 -1 snap_mapper.add_oid found existing snaps mapped on 106:076ecd41:test-rados-api-smithi067-92454-97::foo:31, removing 2023-12-01T17:23:41.732 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:55.513 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.734 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:55.516 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.764 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:55.545 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.778 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:55.538 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.781 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:55.541 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.782 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:55.563 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.785 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:55.567 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.810 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:55.570 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:41.815 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:55.597 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.368 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:56.128 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.369 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:56.129 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.376 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:20:56.137 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.461 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:56.243 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.463 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:56.245 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.471 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:20:56.253 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.512 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:56.294 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.513 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:56.295 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:42.523 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:20:56.305 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.872 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:10.654 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.873 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:10.655 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.903 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:10.684 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.918 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:10.679 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.922 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:10.682 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.927 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:10.708 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.931 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:10.713 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.958 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:10.719 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:23:56.984 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:10.765 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.000 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:15.781 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.002 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:15.784 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.024 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:15.806 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.056 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:15.816 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.058 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:15.819 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.062 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:15.843 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.068 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:15.849 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.076 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:15.836 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:02.090 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:15.870 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:11.988 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:25.769 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:11.991 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:25.773 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:12.018 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:25.799 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:12.050 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:25.810 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:12.051 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:25.833 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:12.051 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:25.812 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:12.053 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:25.834 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:12.074 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:25.835 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:12.089 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:25.870 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.074 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:35.855 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.078 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:35.859 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.103 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:35.885 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.126 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:35.887 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.128 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:35.910 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.129 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:35.911 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.130 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:35.891 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.154 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:35.935 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:22.154 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:35.915 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:26.934 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:40.716 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:26.941 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:40.723 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:26.956 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:40.738 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:26.985 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:40.745 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:26.987 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:40.747 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:26.989 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:40.771 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:26.991 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:40.773 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:27.000 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:40.761 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:27.007 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:40.789 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:41.997 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:55.778 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.001 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:55.783 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.037 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:55.818 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.047 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:55.808 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.050 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:55.810 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.053 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:55.834 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.057 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:55.838 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.082 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:55.843 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.092 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:55.873 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.972 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:56.733 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.973 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:56.734 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:42.982 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:21:56.743 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:43.033 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:56.815 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:43.034 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:56.816 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:43.053 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:21:56.835 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:43.055 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:56.837 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:43.057 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:56.838 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:43.069 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:21:56.851 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.068 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:10.850 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.070 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:10.852 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.103 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:10.885 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.117 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:10.898 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.119 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:10.880 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.122 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:10.904 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.123 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:10.883 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.150 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:10.911 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:57.152 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:10.934 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.188 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:11.948 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.189 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:11.949 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.204 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:11.985 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.205 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:11.986 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.218 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:12.000 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.224 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:11.984 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.378 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:12.160 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.379 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:12.161 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:24:58.394 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:12.176 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.069 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:15.850 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.070 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:15.852 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.083 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:15.865 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.118 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:15.879 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.119 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:15.880 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.120 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:15.901 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.121 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:15.902 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.132 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:15.893 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:02.135 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:15.917 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:10.556 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-12-01T17:25:10.556+0000 7fab6ce17640 -1 calc_pg_upmaps abort due to max <= 0 2023-12-01T17:25:12.139 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:25.920 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.142 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:25.923 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.172 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:25.953 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.205 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:25.965 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.207 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:25.988 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.207 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:25.968 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.219 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:26.001 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.236 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:25.997 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:12.255 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:26.035 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.283 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:36.065 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.285 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:36.066 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.306 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:36.087 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.330 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:36.090 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.332 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:36.093 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.335 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:36.117 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.339 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:36.121 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.351 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:36.112 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:22.360 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:36.141 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.167 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:40.948 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.176 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:40.957 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.194 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:40.976 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.244 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:41.025 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.245 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:41.005 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.245 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:41.006 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.245 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:41.027 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.254 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:41.015 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:27.256 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:41.038 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.406 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:56.187 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.409 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:56.190 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.441 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:22:56.222 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.461 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:56.221 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.462 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:56.222 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.473 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:56.255 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.475 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:56.257 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.485 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:22:56.246 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:42.508 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:22:56.289 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.291 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:11.072 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.295 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:11.077 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.326 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:11.108 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.343 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:11.103 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.345 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:11.127 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.348 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:11.109 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.348 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:11.130 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.376 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:11.136 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:25:57.380 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:11.161 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.309 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:16.090 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.319 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:16.100 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.335 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:16.116 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.356 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:16.117 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.358 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:16.119 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.372 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:16.132 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.394 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:16.176 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.396 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:16.177 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:02.409 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:16.190 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.367 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:26.149 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.369 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:26.151 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.393 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:26.174 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.402 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:26.162 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.404 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:26.165 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.421 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:26.203 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.423 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:26.205 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.426 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:26.187 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:12.504 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:26.285 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.412 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:36.194 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.417 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:36.198 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.450 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:36.232 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.469 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:36.229 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.470 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:36.231 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.481 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:36.262 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.485 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:36.267 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.500 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:36.260 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:22.524 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:36.306 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.508 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:41.290 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.525 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:41.282 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.525 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:41.305 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.526 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:41.286 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.527 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:41.309 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.527 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:41.309 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.595 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:41.355 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.597 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:41.378 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:27.620 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:41.401 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (10849 ms) 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8265 ms) 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6528 ms) 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26535 ms) 2023-12-01T17:26:31.920 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (15732 ms) 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (14266 ms) 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: 00000010 69 73 |is| 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: 00000012 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (13164 ms) 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: require_osd_release = reef 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7901 ms) 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (944146 ms total) 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2023-12-01T17:26:31.921 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1413 ms) 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11795 ms) 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (19929 ms) 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (79 ms) 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (33216 ms total) 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (8250 ms) 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (10159 ms) 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2023-12-01T17:26:31.922 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: waiting for scrub... 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: done waiting 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (27998 ms) 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11911 ms) 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (9607 ms) 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10187 ms) 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11658 ms) 2023-12-01T17:26:31.923 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2023-12-01T17:26:42.496 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:56.277 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.497 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:56.279 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.524 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:56.305 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.541 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:56.302 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.548 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:56.309 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.550 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:56.331 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.554 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:56.336 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.577 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:56.338 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:42.582 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:56.363 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.492 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:57.252 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.493 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:57.253 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.500 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:23:57.261 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.530 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:57.312 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.531 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:57.313 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.539 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:23:57.321 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.582 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:57.364 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.583 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:57.365 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:43.598 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:23:57.380 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.685 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:11.466 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.690 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:11.472 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.719 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:11.501 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.739 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:11.521 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.742 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:11.524 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.752 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:11.513 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.759 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:11.519 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.772 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:11.553 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:26:57.800 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:11.560 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:00.930 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:14.711 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:00.937 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:14.716 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:00.963 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:14.745 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:00.982 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:14.742 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:00.985 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:14.745 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:00.995 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:14.776 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:00.998 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:14.779 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:01.001 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:14.762 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:01.016 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:14.797 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.564 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:16.346 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.570 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:16.351 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.580 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:16.362 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.618 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:16.378 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.619 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:16.379 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.629 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:16.390 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.632 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:16.413 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.635 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:16.417 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:02.650 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:16.431 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.566 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:26.348 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.617 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:26.398 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.645 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:26.426 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.669 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:26.450 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.671 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:26.453 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.678 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:26.438 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.682 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:26.443 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.699 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:26.481 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:12.706 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:26.467 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.781 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:36.563 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.827 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:36.587 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.828 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:36.588 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.842 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:36.623 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.859 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:36.619 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.891 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:36.673 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.912 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:36.694 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.916 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:36.697 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:22.943 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:36.725 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.673 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:41.455 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.675 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:41.456 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.692 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:41.473 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.723 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:41.483 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.725 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:41.486 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.730 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:41.512 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.733 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:41.515 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.743 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:41.504 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:27.748 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:41.530 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.743 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:56.525 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.748 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:56.529 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.778 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:56.560 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.794 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:56.554 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.797 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:56.578 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.800 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:56.581 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.811 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:56.571 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.832 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:56.613 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:42.842 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:56.603 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.808 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:57.569 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.809 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:57.570 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.818 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:24:57.579 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.872 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:57.654 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.873 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:57.655 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.883 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:24:57.665 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.905 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:57.686 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.905 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:57.687 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:27:43.915 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:24:57.697 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:02.961 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:16.743 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:02.967 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:16.749 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:03.002 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:16.783 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:03.033 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:16.814 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:03.036 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:16.796 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:03.043 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:16.804 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:03.051 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:16.832 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:03.076 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:16.837 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:03.087 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:16.868 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.847 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:31.607 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.848 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:31.629 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.848 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:31.608 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.852 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:31.634 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.877 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:31.638 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.891 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:31.673 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.903 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:31.683 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.908 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:31.689 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:17.937 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:31.718 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:22.990 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:36.772 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.058 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:36.840 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.081 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:36.862 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.117 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:36.877 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.118 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:36.879 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.121 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:36.903 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.122 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:36.904 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.135 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:36.895 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:23.138 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:36.919 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.908 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:46.689 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.910 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:46.692 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.923 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:25:46.704 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.955 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:46.716 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.960 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:46.720 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.960 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:46.742 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.964 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:46.745 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.973 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:25:46.734 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:32.979 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:25:46.760 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.082 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:01.863 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.083 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:01.864 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.115 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:01.896 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.137 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:01.918 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.146 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:01.927 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.157 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:01.918 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.162 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:01.922 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.186 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:01.967 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:28:48.189 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:01.950 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.019 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:16.800 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.023 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:16.804 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.044 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:16.826 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.068 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:16.829 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.072 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:16.833 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.073 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:16.855 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.076 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:16.858 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.091 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:16.852 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:03.097 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:16.878 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.864 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:18.646 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.923 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:18.705 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.932 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:18.693 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.936 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:18.717 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.941 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:18.722 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.951 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:18.732 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.966 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:18.726 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.970 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:18.752 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:04.994 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:18.755 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.039 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:21.820 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.045 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:21.826 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.057 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:21.839 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.100 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:21.860 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.101 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:21.861 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.111 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:21.893 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.112 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:21.873 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.112 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:21.894 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:08.126 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:21.907 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.103 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:31.885 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.106 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:31.887 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.133 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:31.915 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.154 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:31.915 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.155 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:31.916 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.165 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:31.947 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.168 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:31.949 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.181 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:31.942 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:18.193 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:31.975 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.421 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:42.203 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.429 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:42.211 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.480 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:42.261 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.499 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:42.260 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.500 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:42.261 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.504 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:42.286 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.507 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:42.289 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.534 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:42.295 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.560 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:42.341 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:28.913 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (9950 ms) 2023-12-01T17:29:28.913 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2023-12-01T17:29:28.913 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14624 ms) 2023-12-01T17:29:28.913 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2023-12-01T17:29:28.913 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10203 ms) 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (14183 ms) 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8782 ms) 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9154 ms) 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: hmm, no HitSet yet 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: ok, hit_set contains 319:602f83fe:::foo:head 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9901 ms) 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,0 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: first is 1701451667 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,0 2023-12-01T17:29:28.914 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,1701451672,1701451673,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,1701451672,1701451673,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,1701451672,1701451673,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,1701451672,1701451673,1701451676,1701451677,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,1701451672,1701451673,1701451676,1701451677,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451667,1701451669,1701451670,1701451672,1701451673,1701451676,1701451677,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: got ls 1701451670,1701451672,1701451673,1701451676,1701451677,1701451679,1701451680,0 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: first now 1701451670, trimmed 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (22334 ms) 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2023-12-01T17:29:28.915 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: foo0 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: verifying foo0 is eventually promoted 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (17239 ms) 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (20066 ms) 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (25240 ms) 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (6834 ms) 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4592 ms) 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (5536 ms) 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4644 ms) 2023-12-01T17:29:28.916 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (273053 ms total) 2023-12-01T17:29:28.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: 2023-12-01T17:29:28.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [----------] Global test environment tear-down 2023-12-01T17:29:28.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1466791 ms total) 2023-12-01T17:29:28.917 INFO:tasks.workunit.client.0.smithi067.stdout: api_tier_pp: [ PASSED ] 77 tests. 2023-12-01T17:29:28.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.921 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92207 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92207 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92404 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92404 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92852 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92852 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.922 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92636 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92636 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92465 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92465 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92305 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92305 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92571 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92571 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93058 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93058 2023-12-01T17:29:28.923 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92321 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92321 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92926 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92926 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92749 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92749 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92178 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92178 2023-12-01T17:29:28.924 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92260 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92260 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92769 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92769 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92169 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92169 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92189 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92189 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93136 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93136 2023-12-01T17:29:28.925 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92420 2023-12-01T17:29:28.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92420 2023-12-01T17:29:28.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:29:28.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93192 2023-12-01T17:29:28.926 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93192 2023-12-01T17:29:29.382 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:43.142 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.383 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:43.143 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.391 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:43.152 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.449 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:43.230 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.450 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:43.231 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.460 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:43.241 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.486 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:43.268 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.487 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:43.269 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:29.499 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:43.281 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.546 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:47.328 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.550 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:47.332 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.561 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:26:47.342 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.595 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:47.376 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.598 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:47.379 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.598 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:47.358 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.599 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:47.359 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.608 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:26:47.390 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:33.609 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:26:47.370 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.457 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:02.238 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.458 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:02.239 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.471 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:02.252 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.508 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:02.289 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.509 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:02.269 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.509 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:02.270 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.510 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:02.291 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.522 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:02.283 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:48.523 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:02.305 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.472 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:07.254 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.474 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:07.255 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.484 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:07.265 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.519 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:07.280 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.520 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:07.281 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.526 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:07.308 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.527 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:07.309 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.529 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:07.290 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:29:53.539 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:07.320 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.760 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:22.541 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.761 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:22.543 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.788 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:22.569 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.828 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:22.609 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.831 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:22.612 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.832 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:22.592 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.836 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:22.596 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.861 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:22.642 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:08.862 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:22.621 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.501 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:23.262 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.502 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:23.263 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.510 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:23.271 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.523 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:23.305 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.524 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:23.306 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.535 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:23.317 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.574 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:23.356 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.575 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:23.357 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:09.583 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:23.365 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.749 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:27.530 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.752 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:27.533 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.762 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:27.543 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.803 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:27.563 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.804 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:27.565 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.814 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:27.574 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.818 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:27.599 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.822 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:27.603 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:13.832 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:27.614 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.792 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:42.573 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.795 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:42.577 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.809 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:42.590 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.843 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:42.625 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.845 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:42.627 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.849 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:42.609 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.850 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:42.611 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.859 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:42.641 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:28.863 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:42.624 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.802 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:47.583 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.805 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:47.587 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.815 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:27:47.596 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.844 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:47.604 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.848 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:47.630 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.854 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:47.635 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.860 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:47.621 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.864 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:27:47.646 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:33.872 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:27:47.633 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.844 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:02.625 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.845 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:02.627 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.859 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:02.640 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.889 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:02.670 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.891 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:02.672 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.900 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:02.660 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.901 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:02.662 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.907 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:02.689 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:48.917 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:02.677 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.853 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:07.634 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.855 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:07.636 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.868 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:07.649 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.899 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:07.659 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.901 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:07.661 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.903 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:07.684 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.904 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:07.686 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.914 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:07.674 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:30:53.916 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:07.697 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.842 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:22.622 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.891 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:22.672 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.907 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:22.689 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.943 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:22.704 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.944 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:22.705 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.959 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:22.720 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.961 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:22.743 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.964 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:22.745 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:08.978 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:22.760 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:13.943 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:27.724 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:13.947 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:27.728 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:13.957 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:27.738 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:13.987 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:27.747 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:13.988 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:27.770 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:13.993 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:27.753 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:13.993 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:27.774 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:14.005 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:27.786 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:14.005 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:27.765 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:23.974 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:37.756 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:23.977 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:37.759 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:23.994 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:37.776 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:24.027 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:37.808 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:24.029 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:37.811 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:24.032 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:37.791 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:24.032 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:37.793 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:24.043 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:37.824 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:24.046 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:37.806 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:28.983 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:42.764 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:28.988 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:42.769 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:29.004 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:42.785 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:29.038 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:42.798 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:29.039 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:42.800 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:29.040 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:42.821 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:29.041 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:42.823 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:29.050 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:42.811 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:29.052 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:42.834 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.016 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:52.797 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.017 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:52.798 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.028 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:28:52.810 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.059 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:52.819 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.063 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:52.824 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.078 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:28:52.838 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.091 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:52.872 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.094 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:52.876 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:39.106 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:28:52.887 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.038 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:02.819 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.042 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:02.824 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.054 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:02.835 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.086 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:02.847 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.088 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:02.870 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.091 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:02.873 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.092 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:02.853 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.106 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:02.865 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:49.108 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:02.890 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.210 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:12.991 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.214 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:12.996 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.229 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:13.010 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.259 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:13.020 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.260 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:13.041 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.260 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:13.021 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.263 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:13.045 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.274 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:13.034 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:31:59.277 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:13.059 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.352 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:17.134 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.357 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:17.138 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.364 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:17.125 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.365 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:17.146 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.367 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:17.128 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.367 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:17.149 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.374 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:17.156 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.384 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:17.166 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:03.385 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:17.145 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.117 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:22.895 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.118 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:22.900 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.135 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:22.917 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.161 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:22.921 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.166 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:22.927 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.178 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:22.939 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.192 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:22.974 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.195 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:22.976 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:09.207 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:22.988 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.141 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:32.922 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.143 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:32.925 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.157 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:32.938 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.191 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:32.951 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.191 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:32.973 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.193 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:32.954 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.195 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:32.977 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.207 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:32.989 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:19.208 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:32.968 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.220 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:48.001 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.225 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:48.006 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.240 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:48.022 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.271 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:48.031 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.271 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:48.053 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.276 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:48.057 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.276 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:48.037 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.290 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:48.072 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:34.292 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:48.052 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.203 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:52.984 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.206 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:52.988 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.219 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:29:53.000 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.253 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:53.013 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.256 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:53.016 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.258 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:53.039 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.259 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:53.041 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.269 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:29:53.030 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:39.272 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:29:53.054 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.245 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:08.026 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.247 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:08.029 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.263 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:08.044 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.298 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:08.059 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.299 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:08.081 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.303 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:08.064 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.317 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:08.099 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.319 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:08.080 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:54.331 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:08.112 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.254 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:13.036 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.256 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:13.038 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.273 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:13.054 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.301 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:13.061 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.304 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:13.065 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.310 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:13.092 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.313 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:13.095 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.319 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:13.080 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:32:59.325 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:13.106 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.295 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:28.077 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.301 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:28.082 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.320 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:28.101 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.352 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:28.113 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.353 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:28.134 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.354 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:28.136 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.355 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:28.116 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.368 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:28.150 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:14.370 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:28.130 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.308 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:33.069 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.309 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:33.070 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.309 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:33.091 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.311 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:33.093 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.324 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:33.085 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.326 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:33.108 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.363 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:33.145 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.367 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:33.148 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:19.379 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:33.161 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.350 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:48.131 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.351 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:48.133 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.364 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:48.145 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.393 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:48.175 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.398 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:48.180 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.401 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:48.161 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.404 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:48.164 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.413 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:48.194 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:34.416 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:48.177 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.360 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:53.142 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.362 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:53.144 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.372 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:30:53.153 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.412 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:53.173 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.413 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:53.194 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.414 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:53.174 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.414 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:53.196 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.423 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:30:53.184 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:39.428 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:30:53.210 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.443 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:08.224 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.447 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:08.229 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.466 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:08.248 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.491 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:08.273 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.495 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:08.277 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.497 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:08.257 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.499 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:08.259 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.513 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:08.295 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:54.515 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:08.276 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.458 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:13.240 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.462 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:13.244 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.474 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:13.256 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.508 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:13.290 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.509 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:13.270 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.511 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:13.271 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.512 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:13.293 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.524 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:13.305 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:33:59.524 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:13.285 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.501 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:28.283 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.503 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:28.285 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.522 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:28.301 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.550 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:28.310 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.554 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:28.314 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.571 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:28.331 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.579 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:28.361 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.581 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:28.363 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:14.596 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:28.377 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.512 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:33.293 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.514 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:33.295 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.524 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:33.305 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.553 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:33.313 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.555 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:33.316 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.559 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:33.341 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.561 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:33.343 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.567 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:33.328 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:19.572 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:33.354 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.553 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:48.335 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.555 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:48.337 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.568 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:48.349 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.600 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:48.381 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.601 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:48.362 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.603 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:48.385 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.605 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:48.365 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.617 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:48.399 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:34.620 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:48.380 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.564 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:53.346 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.565 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:53.347 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.575 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:31:53.357 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.609 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:53.391 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.611 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:53.372 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.613 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:53.394 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.613 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:53.374 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.623 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:31:53.383 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:39.623 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:31:53.405 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.598 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:08.379 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.601 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:08.382 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.613 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:08.394 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.646 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:08.427 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.648 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:08.430 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.651 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:08.411 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.652 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:08.413 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.662 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:08.444 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:54.665 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:08.426 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.613 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:13.395 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.615 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:13.397 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.625 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:13.407 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.653 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:13.413 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.657 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:13.417 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.661 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:13.442 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.663 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:13.445 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.668 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:13.429 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:34:59.675 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:13.456 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.802 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:28.583 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.814 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:28.595 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.829 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:28.611 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.857 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:28.639 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.859 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:28.620 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.861 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:28.642 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.863 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:28.624 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.877 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:28.658 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:14.877 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:28.638 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.440 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:29.201 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.441 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:29.202 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.453 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:29.213 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.495 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:29.276 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.496 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:29.278 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.508 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:29.290 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.524 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:29.305 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.525 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:29.307 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:15.537 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:29.319 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.676 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:33.458 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.682 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:33.463 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.694 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:33.475 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.731 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:33.490 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.731 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:33.513 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.732 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:33.493 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.733 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:33.515 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.743 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:33.525 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:19.744 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:33.505 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.711 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:48.493 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.716 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:48.497 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.731 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:48.513 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.765 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:48.526 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.766 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:48.548 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.767 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:48.527 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.769 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:48.551 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.782 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:48.542 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:34.785 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:48.567 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.750 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:53.532 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.755 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:53.536 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.769 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:32:53.550 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.803 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:53.585 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.806 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:53.587 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.809 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:53.570 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.810 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:53.571 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.819 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:32:53.600 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:39.822 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:32:53.583 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.786 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:08.567 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.839 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:08.600 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.843 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:08.603 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.847 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:08.628 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.875 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:08.636 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.901 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:08.683 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.912 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:08.694 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.915 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:08.697 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:54.941 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:08.722 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.369 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:09.129 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.370 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:09.130 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.378 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:09.139 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.511 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:09.293 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.512 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:09.294 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.524 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:09.306 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.528 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:09.310 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.530 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:09.311 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:35:55.542 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:09.323 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:09.926 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:23.707 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:09.928 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:23.709 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:09.967 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:23.728 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:09.970 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:23.731 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:09.976 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:23.757 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:09.997 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:23.779 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:09.999 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:23.780 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:10.000 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:23.761 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:10.027 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:23.809 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.078 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:28.838 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.124 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:28.906 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.133 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:28.893 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.137 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:28.919 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.138 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:28.920 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.154 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:28.914 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.163 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:28.944 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.195 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:28.977 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:15.219 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:29.000 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.131 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:38.912 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.136 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:38.918 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.159 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:38.940 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.193 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:38.974 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.194 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:38.976 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.196 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:38.957 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.199 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:38.959 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.222 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:39.004 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:25.226 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:38.987 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:27.190 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: Running main() from gmock_main.cc 2023-12-01T17:36:27.190 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2023-12-01T17:36:27.190 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [----------] Global test environment set-up. 2023-12-01T17:36:27.190 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2023-12-01T17:36:27.190 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (1840852 ms) 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3314 ms) 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3484 ms) 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3479 ms) 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3744 ms) 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3638 ms) 2023-12-01T17:36:27.191 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3346 ms) 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2633 ms) 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3604 ms) 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3137 ms) 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3532 ms) 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3234 ms) 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3238 ms) 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2023-12-01T17:36:27.192 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3485 ms) 2023-12-01T17:36:27.193 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (1884720 ms total) 2023-12-01T17:36:27.193 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: 2023-12-01T17:36:27.193 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [----------] Global test environment tear-down 2023-12-01T17:36:27.193 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (1884720 ms total) 2023-12-01T17:36:27.193 INFO:tasks.workunit.client.0.smithi067.stdout: read_operations: [ PASSED ] 14 tests. 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92549 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92549 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93040 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93040 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.196 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93266 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93266 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92444 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92444 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92791 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92791 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93031 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93031 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93173 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93173 2023-12-01T17:36:27.197 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92734 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92734 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92384 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92384 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93241 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93241 2023-12-01T17:36:27.198 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92672 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92672 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92162 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92162 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92231 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92231 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92600 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92600 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.199 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92277 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92277 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92694 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92694 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=92717 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 92717 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93094 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93094 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ for t in "${!pids[@]}" 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ pid=93289 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ wait 93289 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ exit 0 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ cleanup 2023-12-01T17:36:27.200 INFO:tasks.workunit.client.0.smithi067.stderr:+ pkill -P 92157 2023-12-01T17:36:27.215 INFO:tasks.workunit.client.0.smithi067.stderr:+ true 2023-12-01T17:36:27.215 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-01T17:36:27.215 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-12-01T17:36:27.250 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2023-12-01T17:36:27.250 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-12-01T17:36:27.623 DEBUG:teuthology.parallel:result is None 2023-12-01T17:36:27.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2023-12-01T17:36:27.657 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2023-12-01T17:36:27.657 DEBUG:teuthology.orchestra.run.smithi067:> rmdir -- /home/ubuntu/cephtest/mnt.0 2023-12-01T17:36:27.716 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2023-12-01T17:36:27.716 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-01T17:36:27.727 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-12-01T17:36:27.727 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:36:27.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.882+0000 7f70b066f640 10 monclient: get_monmap_and_config 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.882+0000 7f70b066f640 10 monclient: build_initial_monmap 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.882+0000 7f70b066f640 10 monclient: monmap: 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:27.883208+0000 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:27.883208+0000 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:27.883 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 1 Processor -- start 2023-12-01T17:36:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 1 -- start start 2023-12-01T17:36:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient: init 2023-12-01T17:36:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-12-01T17:36:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a813eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:27.884 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient(hunting): picked mon.noname-b con 0x7f70a80a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70a80a8e00 0x7f70a813f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient(hunting): picked mon.noname-a con 0x7f70a80a8e00 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f70a81423b0 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70a80b21f0 con 0x7f70a81423b0 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70a80a9cc0 con 0x7f70a80a8e00 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8142f50 con 0x7f70a80a8a20 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.883+0000 7f70b066f640 10 monclient(hunting): authenticate will time out at 3319.121385s 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70ae40b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a813eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70adc0a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f70a80a8e00 0x7f70a813f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:27.886 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70aec0c640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f70a81423b0 0x7f70a8142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49328/0 (socket says 172.21.15.67:49328) 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70aec0c640 1 -- 172.21.15.67:0/4116462516 learned_addr learned my addr 172.21.15.67:0/4116462516 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70ae40b640 10 monclient(hunting): get_auth_request con 0x7f70a80a8a20 auth_method 0 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70ae40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70ae40b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70ae40b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70adc0a640 10 monclient(hunting): get_auth_request con 0x7f70a80a8e00 auth_method 0 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70adc0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70adc0a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.885+0000 7f70adc0a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.886+0000 7f70ae40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.886+0000 7f70ae40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:27.887 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.886+0000 7f70ae40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.886+0000 7f70adc0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.886+0000 7f70adc0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.886+0000 7f70adc0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 10 monclient(hunting): handle_auth_done global_id 17891 payload 274 2023-12-01T17:36:27.888 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 1 -- 172.21.15.67:0/4116462516 >> v2:172.21.15.67:3300/0 conn(0x7f70a80a8e00 msgr2=0x7f70a813f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 1 --2- 172.21.15.67:0/4116462516 >> v2:172.21.15.67:3300/0 conn(0x7f70a80a8e00 0x7f70a813f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 1 -- 172.21.15.67:0/4116462516 >> v1:172.21.15.67:6789/0 conn(0x7f70a81423b0 legacy=0x7f70a8142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 10 monclient: _finish_hunting 0 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 1 monclient: found mon.noname-b 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.888+0000 7f70ae40b640 1 -- 172.21.15.67:0/4116462516 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a813f930 con 0x7f70a80a8a20 2023-12-01T17:36:27.889 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.890+0000 7f70ae40b640 1 --2- 172.21.15.67:0/4116462516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a813eeb0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f70a40097a0 tx=0x7f70a402b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c78f3629f846faa server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 1 -- 172.21.15.67:0/4116462516 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a402f050 con 0x7f70a80a8a20 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: dump: 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:27.894 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: _finish_auth 0 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: _check_auth_tickets 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 1 -- 172.21.15.67:0/4116462516 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a4031700 con 0x7f70a80a8a20 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70b066f640 5 monclient: authenticate success, global_id 17891 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70b066f640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70b066f640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70b066f640 4 set_mon_vals no callback set 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 1 -- 172.21.15.67:0/4116462516 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4031a90 con 0x7f70a80a8a20 2023-12-01T17:36:27.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.894+0000 7f70ad409640 10 monclient: dump: 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:27.896 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 10 monclient: shutdown 2023-12-01T17:36:27.899 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 -- 172.21.15.67:0/4116462516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 msgr2=0x7f70a813eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 --2- 172.21.15.67:0/4116462516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a813eeb0 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f70a40097a0 tx=0x7f70a402b7a0 comp rx=0 tx=0).stop 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 -- 172.21.15.67:0/4116462516 shutdown_connections 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 --2- 172.21.15.67:0/4116462516 >> v2:172.21.15.67:3300/0 conn(0x7f70a80a8e00 0x7f70a813f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 --2- 172.21.15.67:0/4116462516 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a813eeb0 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 -- 172.21.15.67:0/4116462516 >> 172.21.15.67:0/4116462516 conn(0x7f70a80b7870 msgr2=0x7f70a80b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 -- 172.21.15.67:0/4116462516 shutdown_connections 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.895+0000 7f70b066f640 1 -- 172.21.15.67:0/4116462516 wait complete. 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 10 monclient: build_initial_monmap 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 10 monclient: monmap: 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:27.900 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 1 librados: starting msgr at 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 1 librados: starting objecter 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 1 Processor -- start 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 1 -- start start 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 1 librados: setting wanted keys 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 1 librados: calling monclient init 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.896+0000 7f70b066f640 10 monclient: init 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:27.901 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a8136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): picked mon.noname-c con 0x7f70a80a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a80a8e00 0x7f70a8144860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): picked mon.noname-b con 0x7f70a80a8e00 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a81423b0 0x7f70a8143390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): picked mon.noname-a con 0x7f70a81423b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8145200 con 0x7f70a81423b0 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70a8145690 con 0x7f70a80a8e00 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70a81462a0 con 0x7f70a80a8a20 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:27.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.897+0000 7f70b066f640 10 monclient(hunting): authenticate will time out at 3319.134893s 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70ae40b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a8136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70adc0a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a80a8e00 0x7f70a8144860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70aec0c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a81423b0 0x7f70a8143390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70adc0a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a80a8e00 0x7f70a8144860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47668/0 (socket says 172.21.15.67:47668) 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70adc0a640 1 -- 172.21.15.67:0/3368454549 learned_addr learned my addr 172.21.15.67:0/3368454549 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70adc0a640 10 monclient(hunting): get_auth_request con 0x7f70a80a8e00 auth_method 0 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70adc0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70adc0a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70adc0a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70aec0c640 10 monclient(hunting): get_auth_request con 0x7f70a81423b0 auth_method 0 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70aec0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70aec0c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70aec0c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70ae40b640 10 monclient(hunting): get_auth_request con 0x7f70a80a8a20 auth_method 0 2023-12-01T17:36:27.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70ae40b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70ae40b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.898+0000 7f70ae40b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70adc0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70adc0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70adc0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70aec0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70aec0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70aec0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70ae40b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70ae40b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:27.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.899+0000 7f70ae40b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 10 monclient(hunting): handle_auth_done global_id 23134 payload 824 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 1 -- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 msgr2=0x7f70a8136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a8136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 1 -- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a81423b0 msgr2=0x7f70a8143390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a81423b0 0x7f70a8143390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 10 monclient: _finish_hunting 0 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 1 monclient: found mon.noname-b 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:27.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.901+0000 7f70adc0a640 1 -- 172.21.15.67:0/3368454549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a4009400 con 0x7f70a80a8e00 2023-12-01T17:36:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.903+0000 7f70adc0a640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a80a8e00 0x7f70a8144860 secure :-1 s=READY pgs=5877 cs=0 l=1 rev1=1 crypto rx=0x7f709800e980 tx=0x7f709800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.903+0000 7f709effd640 10 client.?.objecter ms_handle_connect 0x7f70a80a8e00 2023-12-01T17:36:27.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.903+0000 7f709effd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 1 -- 172.21.15.67:0/3368454549 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7098009670 con 0x7f70a80a8e00 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: dump: 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:27.907 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: _finish_auth 0 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 1 -- 172.21.15.67:0/3368454549 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709800ca00 con 0x7f70a80a8e00 2023-12-01T17:36:27.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 5 monclient: authenticate success, global_id 23134 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 10 monclient: _renew_subs 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70a8144da0 con 0x7f70a80a8e00 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 1 -- 172.21.15.67:0/3368454549 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f709800cd90 con 0x7f70a80a8e00 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f709effd640 10 monclient: dump: 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:27.909 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70acc08640 4 set_mon_vals no callback set 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 10 client.23134.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 10 monclient: _renew_subs 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70a8143ab0 con 0x7f70a80a8e00 2023-12-01T17:36:27.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.907+0000 7f70b066f640 1 librados: init done 2023-12-01T17:36:27.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.908+0000 7f70b066f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:27.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.908+0000 7f70b066f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:27.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.908+0000 7f70b066f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:27.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.908+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a8145370 con 0x7f70a80a8e00 2023-12-01T17:36:27.926 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.926+0000 7f709effd640 1 -- 172.21.15.67:0/3368454549 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f7098013470 con 0x7f70a80a8e00 2023-12-01T17:36:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.927+0000 7f709effd640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f708c06c820 0x7f708c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.927+0000 7f70ae40b640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f708c06c820 0x7f708c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.927+0000 7f70ae40b640 10 monclient: get_auth_request con 0x7f708c06c820 auth_method 0 2023-12-01T17:36:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.927+0000 7f709effd640 10 client.23134.objecter ms_handle_connect 0x7f708c06c820 2023-12-01T17:36:27.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:27.927+0000 7f70ae40b640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f708c06c820 0x7f708c06ece0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f70a4035000 tx=0x7f70a4008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:28.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.146+0000 7f709effd640 1 -- 172.21.15.67:0/3368454549 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1006..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f7098058db0 con 0x7f70a80a8e00 2023-12-01T17:36:28.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.146+0000 7f709effd640 10 client.23134.objecter ms_dispatch 0x7f70a8000b90 osd_map(1099..1099 src has 1006..1099) v4 2023-12-01T17:36:28.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.146+0000 7f709effd640 3 client.23134.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:28.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.146+0000 7f709effd640 3 client.23134.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:28.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.146+0000 7f709effd640 20 client.23134.objecter dump_active .. 0 homeless 2023-12-01T17:36:28.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.155+0000 7f709effd640 1 -- 172.21.15.67:0/3368454549 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f70980550f0 con 0x7f70a80a8e00 2023-12-01T17:36:28.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.155+0000 7f709effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:28.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.155+0000 7f709effd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:28.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.378+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70a80aacb0 con 0x7f708c06c820 2023-12-01T17:36:28.383 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.383+0000 7f709effd640 1 -- 172.21.15.67:0/3368454549 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151416 (secure 0 0 0) 0x7f70a4042070 con 0x7f708c06c820 2023-12-01T17:36:28.384 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:36:28.385 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 10 librados: watch_flush enter 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 10 librados: watch_flush exit 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 20 client.23134.objecter shutdown clearing up homeless session... 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 10 client.23134.objecter successfully canceled tick 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f708c06c820 msgr2=0x7f708c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f708c06c820 0x7f708c06ece0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f70a4035000 tx=0x7f70a4008030 comp rx=0 tx=0).stop 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 10 monclient: shutdown 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a80a8e00 msgr2=0x7f70a8144860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a80a8e00 0x7f70a8144860 secure :-1 s=READY pgs=5877 cs=0 l=1 rev1=1 crypto rx=0x7f709800e980 tx=0x7f709800ee40 comp rx=0 tx=0).stop 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 shutdown_connections 2023-12-01T17:36:28.387 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a81423b0 0x7f70a8143390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f70a80a8e00 0x7f70a8144860 unknown :-1 s=CLOSED pgs=5877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f708c06c820 0x7f708c06ece0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 --2- 172.21.15.67:0/3368454549 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f70a80a8a20 0x7f70a8136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 >> 172.21.15.67:0/3368454549 conn(0x7f70a80b7870 msgr2=0x7f70a8142840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 shutdown_connections 2023-12-01T17:36:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 -- 172.21.15.67:0/3368454549 wait complete. 2023-12-01T17:36:28.388 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.386+0000 7f70b066f640 1 librados: shutdown 2023-12-01T17:36:28.397 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":2080,"stamp":"2023-12-01T17:36:27.133808+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":577168,"kb_used_data":60072,"kb_used_omap":12,"kb_used_meta":513139,"kb_avail":754397552,"statfs":{"total":773094113280,"available":772503093248,"internally_reserved":0,"allocated":61513728,"data_stored":34357404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":525454896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":395,"apply_latency_ms":395,"commit_latency_ns":395000000,"apply_latency_ns":395000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.495027"},"pg_stats":[{"pgid":"22.6","version":"0'0","reported_seq":1045,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.677997+0000","last_change":"2023-12-01T17:05:14.287690+0000","last_active":"2023-12-01T17:36:22.677997+0000","last_peered":"2023-12-01T17:36:22.677997+0000","last_clean":"2023-12-01T17:36:22.677997+0000","last_became_active":"2023-12-01T17:05:14.257152+0000","last_became_peered":"2023-12-01T17:05:14.257152+0000","last_unstale":"2023-12-01T17:36:22.677997+0000","last_undegraded":"2023-12-01T17:36:22.677997+0000","last_fullsized":"2023-12-01T17:36:22.677997+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:59:48.591916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1053,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.536581+0000","last_change":"2023-12-01T17:05:21.303515+0000","last_active":"2023-12-01T17:36:21.536581+0000","last_peered":"2023-12-01T17:36:21.536581+0000","last_clean":"2023-12-01T17:36:21.536581+0000","last_became_active":"2023-12-01T17:05:21.296791+0000","last_became_peered":"2023-12-01T17:05:21.296791+0000","last_unstale":"2023-12-01T17:36:21.536581+0000","last_undegraded":"2023-12-01T17:36:21.536581+0000","last_fullsized":"2023-12-01T17:36:21.536581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:51:36.558228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1051,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.476084+0000","last_change":"2023-12-01T17:05:13.270729+0000","last_active":"2023-12-01T17:36:21.476084+0000","last_peered":"2023-12-01T17:36:21.476084+0000","last_clean":"2023-12-01T17:36:21.476084+0000","last_became_active":"2023-12-01T17:05:13.265267+0000","last_became_peered":"2023-12-01T17:05:13.265267+0000","last_unstale":"2023-12-01T17:36:21.476084+0000","last_undegraded":"2023-12-01T17:36:21.476084+0000","last_fullsized":"2023-12-01T17:36:21.476084+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:57:02.423320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1049,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.596184+0000","last_change":"2023-12-01T17:05:38.668209+0000","last_active":"2023-12-01T17:36:22.596184+0000","last_peered":"2023-12-01T17:36:22.596184+0000","last_clean":"2023-12-01T17:36:22.596184+0000","last_became_active":"2023-12-01T17:05:38.651540+0000","last_became_peered":"2023-12-01T17:05:38.651540+0000","last_unstale":"2023-12-01T17:36:22.596184+0000","last_undegraded":"2023-12-01T17:36:22.596184+0000","last_fullsized":"2023-12-01T17:36:22.596184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:43:47.235945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1058,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.693835+0000","last_change":"2023-12-01T17:05:45.108671+0000","last_active":"2023-12-01T17:36:21.693835+0000","last_peered":"2023-12-01T17:36:21.693835+0000","last_clean":"2023-12-01T17:36:21.693835+0000","last_became_active":"2023-12-01T17:05:45.102496+0000","last_became_peered":"2023-12-01T17:05:45.102496+0000","last_unstale":"2023-12-01T17:36:21.693835+0000","last_undegraded":"2023-12-01T17:36:21.693835+0000","last_fullsized":"2023-12-01T17:36:21.693835+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:35:01.639680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.590006+0000","last_change":"2023-12-01T17:05:21.134670+0000","last_active":"2023-12-01T17:36:22.590006+0000","last_peered":"2023-12-01T17:36:22.590006+0000","last_clean":"2023-12-01T17:36:22.590006+0000","last_became_active":"2023-12-01T17:05:21.112377+0000","last_became_peered":"2023-12-01T17:05:21.112377+0000","last_unstale":"2023-12-01T17:36:22.590006+0000","last_undegraded":"2023-12-01T17:36:22.590006+0000","last_fullsized":"2023-12-01T17:36:22.590006+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:40:39.378498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1049,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.465491+0000","last_change":"2023-12-01T17:05:12.338713+0000","last_active":"2023-12-01T17:36:21.465491+0000","last_peered":"2023-12-01T17:36:21.465491+0000","last_clean":"2023-12-01T17:36:21.465491+0000","last_became_active":"2023-12-01T17:05:12.325443+0000","last_became_peered":"2023-12-01T17:05:12.325443+0000","last_unstale":"2023-12-01T17:36:21.465491+0000","last_undegraded":"2023-12-01T17:36:21.465491+0000","last_fullsized":"2023-12-01T17:36:21.465491+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:16:52.693300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.577087+0000","last_change":"2023-12-01T17:05:35.977136+0000","last_active":"2023-12-01T17:36:22.577087+0000","last_peered":"2023-12-01T17:36:22.577087+0000","last_clean":"2023-12-01T17:36:22.577087+0000","last_became_active":"2023-12-01T17:05:35.971286+0000","last_became_peered":"2023-12-01T17:05:35.971286+0000","last_unstale":"2023-12-01T17:36:22.577087+0000","last_undegraded":"2023-12-01T17:36:22.577087+0000","last_fullsized":"2023-12-01T17:36:22.577087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:40:27.793735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1046,"reported_epoch":1096,"state":"active+clean","last_fresh":"2023-12-01T17:36:25.149276+0000","last_change":"2023-12-01T17:05:33.458007+0000","last_active":"2023-12-01T17:36:25.149276+0000","last_peered":"2023-12-01T17:36:25.149276+0000","last_clean":"2023-12-01T17:36:25.149276+0000","last_became_active":"2023-12-01T17:05:33.448448+0000","last_became_peered":"2023-12-01T17:05:33.448448+0000","last_unstale":"2023-12-01T17:36:25.149276+0000","last_undegraded":"2023-12-01T17:36:25.149276+0000","last_fullsized":"2023-12-01T17:36:25.149276+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:03:13.718758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1051,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.606030+0000","last_change":"2023-12-01T17:05:36.335708+0000","last_active":"2023-12-01T17:36:22.606030+0000","last_peered":"2023-12-01T17:36:22.606030+0000","last_clean":"2023-12-01T17:36:22.606030+0000","last_became_active":"2023-12-01T17:05:36.329117+0000","last_became_peered":"2023-12-01T17:05:36.329117+0000","last_unstale":"2023-12-01T17:36:22.606030+0000","last_undegraded":"2023-12-01T17:36:22.606030+0000","last_fullsized":"2023-12-01T17:36:22.606030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T00:46:41.192583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1049,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.477571+0000","last_change":"2023-12-01T17:05:11.457400+0000","last_active":"2023-12-01T17:36:21.477571+0000","last_peered":"2023-12-01T17:36:21.477571+0000","last_clean":"2023-12-01T17:36:21.477571+0000","last_became_active":"2023-12-01T17:05:11.452081+0000","last_became_peered":"2023-12-01T17:05:11.452081+0000","last_unstale":"2023-12-01T17:36:21.477571+0000","last_undegraded":"2023-12-01T17:36:21.477571+0000","last_fullsized":"2023-12-01T17:36:21.477571+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:10:31.762918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1051,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.526103+0000","last_change":"2023-12-01T17:05:36.195013+0000","last_active":"2023-12-01T17:36:21.526103+0000","last_peered":"2023-12-01T17:36:21.526103+0000","last_clean":"2023-12-01T17:36:21.526103+0000","last_became_active":"2023-12-01T17:05:36.185378+0000","last_became_peered":"2023-12-01T17:05:36.185378+0000","last_unstale":"2023-12-01T17:36:21.526103+0000","last_undegraded":"2023-12-01T17:36:21.526103+0000","last_fullsized":"2023-12-01T17:36:21.526103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:38:11.192241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":1159,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.489678+0000","last_change":"2023-12-01T17:03:54.894752+0000","last_active":"2023-12-01T17:36:22.489678+0000","last_peered":"2023-12-01T17:36:22.489678+0000","last_clean":"2023-12-01T17:36:22.489678+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:36:22.489678+0000","last_undegraded":"2023-12-01T17:36:22.489678+0000","last_fullsized":"2023-12-01T17:36:22.489678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_clean_scrub_stamp":"2023-12-01T17:03:51.457677+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:12:02.023301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1049,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.660940+0000","last_change":"2023-12-01T17:04:01.495183+0000","last_active":"2023-12-01T17:36:22.660940+0000","last_peered":"2023-12-01T17:36:22.660940+0000","last_clean":"2023-12-01T17:36:22.660940+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:36:22.660940+0000","last_undegraded":"2023-12-01T17:36:22.660940+0000","last_fullsized":"2023-12-01T17:36:22.660940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:13:19.704857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1044,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.563641+0000","last_change":"2023-12-01T17:05:15.674793+0000","last_active":"2023-12-01T17:36:22.563641+0000","last_peered":"2023-12-01T17:36:22.563641+0000","last_clean":"2023-12-01T17:36:22.563641+0000","last_became_active":"2023-12-01T17:05:15.659631+0000","last_became_peered":"2023-12-01T17:05:15.659631+0000","last_unstale":"2023-12-01T17:36:22.563641+0000","last_undegraded":"2023-12-01T17:36:22.563641+0000","last_fullsized":"2023-12-01T17:36:22.563641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:04:05.554372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1051,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.638318+0000","last_change":"2023-12-01T17:04:01.146575+0000","last_active":"2023-12-01T17:36:22.638318+0000","last_peered":"2023-12-01T17:36:22.638318+0000","last_clean":"2023-12-01T17:36:22.638318+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:36:22.638318+0000","last_undegraded":"2023-12-01T17:36:22.638318+0000","last_fullsized":"2023-12-01T17:36:22.638318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:18:50.459254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1047,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.513553+0000","last_change":"2023-12-01T17:05:11.335421+0000","last_active":"2023-12-01T17:36:22.513553+0000","last_peered":"2023-12-01T17:36:22.513553+0000","last_clean":"2023-12-01T17:36:22.513553+0000","last_became_active":"2023-12-01T17:05:11.329382+0000","last_became_peered":"2023-12-01T17:05:11.329382+0000","last_unstale":"2023-12-01T17:36:22.513553+0000","last_undegraded":"2023-12-01T17:36:22.513553+0000","last_fullsized":"2023-12-01T17:36:22.513553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:01:09.667175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1057,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.528202+0000","last_change":"2023-12-01T17:04:01.107520+0000","last_active":"2023-12-01T17:36:21.528202+0000","last_peered":"2023-12-01T17:36:21.528202+0000","last_clean":"2023-12-01T17:36:21.528202+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:36:21.528202+0000","last_undegraded":"2023-12-01T17:36:21.528202+0000","last_fullsized":"2023-12-01T17:36:21.528202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:47:47.456810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1048,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.479275+0000","last_change":"2023-12-01T17:05:16.493438+0000","last_active":"2023-12-01T17:36:21.479275+0000","last_peered":"2023-12-01T17:36:21.479275+0000","last_clean":"2023-12-01T17:36:21.479275+0000","last_became_active":"2023-12-01T17:05:16.481672+0000","last_became_peered":"2023-12-01T17:05:16.481672+0000","last_unstale":"2023-12-01T17:36:21.479275+0000","last_undegraded":"2023-12-01T17:36:21.479275+0000","last_fullsized":"2023-12-01T17:36:21.479275+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:22:21.064006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1050,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.647330+0000","last_change":"2023-12-01T17:04:01.438358+0000","last_active":"2023-12-01T17:36:22.647330+0000","last_peered":"2023-12-01T17:36:22.647330+0000","last_clean":"2023-12-01T17:36:22.647330+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:36:22.647330+0000","last_undegraded":"2023-12-01T17:36:22.647330+0000","last_fullsized":"2023-12-01T17:36:22.647330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:45:24.963538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1050,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.533853+0000","last_change":"2023-12-01T17:05:14.953876+0000","last_active":"2023-12-01T17:36:21.533853+0000","last_peered":"2023-12-01T17:36:21.533853+0000","last_clean":"2023-12-01T17:36:21.533853+0000","last_became_active":"2023-12-01T17:05:14.948506+0000","last_became_peered":"2023-12-01T17:05:14.948506+0000","last_unstale":"2023-12-01T17:36:21.533853+0000","last_undegraded":"2023-12-01T17:36:21.533853+0000","last_fullsized":"2023-12-01T17:36:21.533853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:52.139356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1055,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.474596+0000","last_change":"2023-12-01T17:04:01.467571+0000","last_active":"2023-12-01T17:36:21.474596+0000","last_peered":"2023-12-01T17:36:21.474596+0000","last_clean":"2023-12-01T17:36:21.474596+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:36:21.474596+0000","last_undegraded":"2023-12-01T17:36:21.474596+0000","last_fullsized":"2023-12-01T17:36:21.474596+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:41:15.693504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1047,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.486427+0000","last_change":"2023-12-01T17:05:11.984847+0000","last_active":"2023-12-01T17:36:22.486427+0000","last_peered":"2023-12-01T17:36:22.486427+0000","last_clean":"2023-12-01T17:36:22.486427+0000","last_became_active":"2023-12-01T17:05:11.693016+0000","last_became_peered":"2023-12-01T17:05:11.693016+0000","last_unstale":"2023-12-01T17:36:22.486427+0000","last_undegraded":"2023-12-01T17:36:22.486427+0000","last_fullsized":"2023-12-01T17:36:22.486427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:40:29.327298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1050,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.681781+0000","last_change":"2023-12-01T17:04:01.201853+0000","last_active":"2023-12-01T17:36:22.681781+0000","last_peered":"2023-12-01T17:36:22.681781+0000","last_clean":"2023-12-01T17:36:22.681781+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:36:22.681781+0000","last_undegraded":"2023-12-01T17:36:22.681781+0000","last_fullsized":"2023-12-01T17:36:22.681781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:38:19.219620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1063,"reported_epoch":1095,"state":"active+clean","last_fresh":"2023-12-01T17:36:23.675866+0000","last_change":"2023-12-01T17:05:12.792097+0000","last_active":"2023-12-01T17:36:23.675866+0000","last_peered":"2023-12-01T17:36:23.675866+0000","last_clean":"2023-12-01T17:36:23.675866+0000","last_became_active":"2023-12-01T17:05:12.768446+0000","last_became_peered":"2023-12-01T17:05:12.768446+0000","last_unstale":"2023-12-01T17:36:23.675866+0000","last_undegraded":"2023-12-01T17:36:23.675866+0000","last_fullsized":"2023-12-01T17:36:23.675866+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:17:04.335014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1063,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.719744+0000","last_change":"2023-12-01T17:04:01.536883+0000","last_active":"2023-12-01T17:36:21.719744+0000","last_peered":"2023-12-01T17:36:21.719744+0000","last_clean":"2023-12-01T17:36:21.719744+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:36:21.719744+0000","last_undegraded":"2023-12-01T17:36:21.719744+0000","last_fullsized":"2023-12-01T17:36:21.719744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:42:47.593805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1053,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.522414+0000","last_change":"2023-12-01T17:05:24.990862+0000","last_active":"2023-12-01T17:36:21.522414+0000","last_peered":"2023-12-01T17:36:21.522414+0000","last_clean":"2023-12-01T17:36:21.522414+0000","last_became_active":"2023-12-01T17:05:24.924547+0000","last_became_peered":"2023-12-01T17:05:24.924547+0000","last_unstale":"2023-12-01T17:36:21.522414+0000","last_undegraded":"2023-12-01T17:36:21.522414+0000","last_fullsized":"2023-12-01T17:36:21.522414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:58.899609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.698718+0000","last_change":"2023-12-01T17:04:01.476480+0000","last_active":"2023-12-01T17:36:22.698718+0000","last_peered":"2023-12-01T17:36:22.698718+0000","last_clean":"2023-12-01T17:36:22.698718+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:36:22.698718+0000","last_undegraded":"2023-12-01T17:36:22.698718+0000","last_fullsized":"2023-12-01T17:36:22.698718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:26:16.264231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1059,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.695845+0000","last_change":"2023-12-01T17:05:28.757137+0000","last_active":"2023-12-01T17:36:21.695845+0000","last_peered":"2023-12-01T17:36:21.695845+0000","last_clean":"2023-12-01T17:36:21.695845+0000","last_became_active":"2023-12-01T17:05:28.748541+0000","last_became_peered":"2023-12-01T17:05:28.748541+0000","last_unstale":"2023-12-01T17:36:21.695845+0000","last_undegraded":"2023-12-01T17:36:21.695845+0000","last_fullsized":"2023-12-01T17:36:21.695845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:15:33.293632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1","version":"62'7","reported_seq":1064,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.578985+0000","last_change":"2023-12-01T17:05:38.783874+0000","last_active":"2023-12-01T17:36:22.578985+0000","last_peered":"2023-12-01T17:36:22.578985+0000","last_clean":"2023-12-01T17:36:22.578985+0000","last_became_active":"2023-12-01T17:05:38.775069+0000","last_became_peered":"2023-12-01T17:05:38.775069+0000","last_unstale":"2023-12-01T17:36:22.578985+0000","last_undegraded":"2023-12-01T17:36:22.578985+0000","last_fullsized":"2023-12-01T17:36:22.578985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:09:31.642602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1052,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.565789+0000","last_change":"2023-12-01T17:05:17.531869+0000","last_active":"2023-12-01T17:36:22.565789+0000","last_peered":"2023-12-01T17:36:22.565789+0000","last_clean":"2023-12-01T17:36:22.565789+0000","last_became_active":"2023-12-01T17:05:17.447969+0000","last_became_peered":"2023-12-01T17:05:17.447969+0000","last_unstale":"2023-12-01T17:36:22.565789+0000","last_undegraded":"2023-12-01T17:36:22.565789+0000","last_fullsized":"2023-12-01T17:36:22.565789+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:20:05.487052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.496765+0000","last_change":"2023-12-01T17:05:12.380756+0000","last_active":"2023-12-01T17:36:22.496765+0000","last_peered":"2023-12-01T17:36:22.496765+0000","last_clean":"2023-12-01T17:36:22.496765+0000","last_became_active":"2023-12-01T17:05:12.334114+0000","last_became_peered":"2023-12-01T17:05:12.334114+0000","last_unstale":"2023-12-01T17:36:22.496765+0000","last_undegraded":"2023-12-01T17:36:22.496765+0000","last_fullsized":"2023-12-01T17:36:22.496765+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T00:52:19.170433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1065,"reported_epoch":1095,"state":"active+clean","last_fresh":"2023-12-01T17:36:23.690252+0000","last_change":"2023-12-01T17:05:39.346000+0000","last_active":"2023-12-01T17:36:23.690252+0000","last_peered":"2023-12-01T17:36:23.690252+0000","last_clean":"2023-12-01T17:36:23.690252+0000","last_became_active":"2023-12-01T17:05:39.340484+0000","last_became_peered":"2023-12-01T17:05:39.340484+0000","last_unstale":"2023-12-01T17:36:23.690252+0000","last_undegraded":"2023-12-01T17:36:23.690252+0000","last_fullsized":"2023-12-01T17:36:23.690252+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:50:46.835583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.517739+0000","last_change":"2023-12-01T17:05:58.777840+0000","last_active":"2023-12-01T17:36:22.517739+0000","last_peered":"2023-12-01T17:36:22.517739+0000","last_clean":"2023-12-01T17:36:22.517739+0000","last_became_active":"2023-12-01T17:05:58.772230+0000","last_became_peered":"2023-12-01T17:05:58.772230+0000","last_unstale":"2023-12-01T17:36:22.517739+0000","last_undegraded":"2023-12-01T17:36:22.517739+0000","last_fullsized":"2023-12-01T17:36:22.517739+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:05:58.781340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1044,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.667139+0000","last_change":"2023-12-01T17:05:11.225296+0000","last_active":"2023-12-01T17:36:22.667139+0000","last_peered":"2023-12-01T17:36:22.667139+0000","last_clean":"2023-12-01T17:36:22.667139+0000","last_became_active":"2023-12-01T17:05:11.209283+0000","last_became_peered":"2023-12-01T17:05:11.209283+0000","last_unstale":"2023-12-01T17:36:22.667139+0000","last_undegraded":"2023-12-01T17:36:22.667139+0000","last_fullsized":"2023-12-01T17:36:22.667139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:39:18.388303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.566053+0000","last_change":"2023-12-01T17:05:17.271434+0000","last_active":"2023-12-01T17:36:22.566053+0000","last_peered":"2023-12-01T17:36:22.566053+0000","last_clean":"2023-12-01T17:36:22.566053+0000","last_became_active":"2023-12-01T17:05:17.215461+0000","last_became_peered":"2023-12-01T17:05:17.215461+0000","last_unstale":"2023-12-01T17:36:22.566053+0000","last_undegraded":"2023-12-01T17:36:22.566053+0000","last_fullsized":"2023-12-01T17:36:22.566053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:14:02.813273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1044,"reported_epoch":1096,"state":"active+clean","last_fresh":"2023-12-01T17:36:25.151563+0000","last_change":"2023-12-01T17:05:21.697504+0000","last_active":"2023-12-01T17:36:25.151563+0000","last_peered":"2023-12-01T17:36:25.151563+0000","last_clean":"2023-12-01T17:36:25.151563+0000","last_became_active":"2023-12-01T17:05:21.668214+0000","last_became_peered":"2023-12-01T17:05:21.668214+0000","last_unstale":"2023-12-01T17:36:25.151563+0000","last_undegraded":"2023-12-01T17:36:25.151563+0000","last_fullsized":"2023-12-01T17:36:25.151563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:07:59.125552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1060,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.688026+0000","last_change":"2023-12-01T17:05:46.022392+0000","last_active":"2023-12-01T17:36:21.688026+0000","last_peered":"2023-12-01T17:36:21.688026+0000","last_clean":"2023-12-01T17:36:21.688026+0000","last_became_active":"2023-12-01T17:05:46.016610+0000","last_became_peered":"2023-12-01T17:05:46.016610+0000","last_unstale":"2023-12-01T17:36:21.688026+0000","last_undegraded":"2023-12-01T17:36:21.688026+0000","last_fullsized":"2023-12-01T17:36:21.688026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:47:36.167822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.636608+0000","last_change":"2023-12-01T17:05:37.365267+0000","last_active":"2023-12-01T17:36:22.636608+0000","last_peered":"2023-12-01T17:36:22.636608+0000","last_clean":"2023-12-01T17:36:22.636608+0000","last_became_active":"2023-12-01T17:05:37.358444+0000","last_became_peered":"2023-12-01T17:05:37.358444+0000","last_unstale":"2023-12-01T17:36:22.636608+0000","last_undegraded":"2023-12-01T17:36:22.636608+0000","last_fullsized":"2023-12-01T17:36:22.636608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:48:13.502545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1050,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.471869+0000","last_change":"2023-12-01T17:05:13.509587+0000","last_active":"2023-12-01T17:36:21.471869+0000","last_peered":"2023-12-01T17:36:21.471869+0000","last_clean":"2023-12-01T17:36:21.471869+0000","last_became_active":"2023-12-01T17:05:13.499129+0000","last_became_peered":"2023-12-01T17:05:13.499129+0000","last_unstale":"2023-12-01T17:36:21.471869+0000","last_undegraded":"2023-12-01T17:36:21.471869+0000","last_fullsized":"2023-12-01T17:36:21.471869+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:25:34.541978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.649060+0000","last_change":"2023-12-01T17:05:37.748101+0000","last_active":"2023-12-01T17:36:22.649060+0000","last_peered":"2023-12-01T17:36:22.649060+0000","last_clean":"2023-12-01T17:36:22.649060+0000","last_became_active":"2023-12-01T17:05:37.740514+0000","last_became_peered":"2023-12-01T17:05:37.740514+0000","last_unstale":"2023-12-01T17:36:22.649060+0000","last_undegraded":"2023-12-01T17:36:22.649060+0000","last_fullsized":"2023-12-01T17:36:22.649060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T04:17:18.757175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444421,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71768,"kb_used_data":8028,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300072,"statfs":{"total":96636764160,"available":96563273728,"internally_reserved":0,"allocated":8220672,"data_stored":4735665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":20.553999999999998,"5min":12.125999999999999,"15min":15.801},"min":{"1min":6.5620000000000003,"5min":4.5899999999999999,"15min":4.5899999999999999},"max":{"1min":127.741,"5min":127.741,"15min":150.279},"last":8.2080000000000002},{"interface":"front","average":{"1min":29.547000000000001,"5min":14.099,"15min":17.922999999999998},"min":{"1min":8.5419999999999998,"5min":4.7229999999999999,"15min":4.7229999999999999},"max":{"1min":184.30600000000001,"5min":184.30600000000001,"15min":184.30600000000001},"last":10.763}]},{"osd":2,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":18.321999999999999,"5min":12.057,"15min":16.654},"min":{"1min":6.5709999999999997,"5min":5.4610000000000003,"15min":5.4610000000000003},"max":{"1min":99.156000000000006,"5min":99.156000000000006,"15min":208.18199999999999},"last":8.8569999999999993},{"interface":"front","average":{"1min":34.377000000000002,"5min":16.986999999999998,"15min":19.702000000000002},"min":{"1min":7.9260000000000002,"5min":6.8769999999999998,"15min":6.2030000000000003},"max":{"1min":216.785,"5min":216.785,"15min":216.785},"last":12.122999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":28.09,"5min":16.707999999999998,"15min":21.498000000000001},"min":{"1min":8.9960000000000004,"5min":7.8550000000000004,"15min":7.8550000000000004},"max":{"1min":243.88999999999999,"5min":243.88999999999999,"15min":276.07400000000001},"last":16.044},{"interface":"front","average":{"1min":30.187000000000001,"5min":19.353999999999999,"15min":24.324000000000002},"min":{"1min":13.336,"5min":10.973000000000001,"15min":7.2649999999999997},"max":{"1min":239.40000000000001,"5min":239.40000000000001,"15min":252.43299999999999},"last":36.654000000000003}]},{"osd":4,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":31.789000000000001,"5min":16.448,"15min":17.77},"min":{"1min":8.0429999999999993,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":230.511,"5min":230.511,"15min":230.511},"last":11.507999999999999},{"interface":"front","average":{"1min":33.975000000000001,"5min":17.349,"15min":21.777999999999999},"min":{"1min":7.6379999999999999,"5min":7.3979999999999997,"15min":7.3979999999999997},"max":{"1min":241.28100000000001,"5min":241.28100000000001,"15min":241.28100000000001},"last":9.6859999999999999}]},{"osd":5,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":27.834,"5min":16.713999999999999,"15min":18.786000000000001},"min":{"1min":6.859,"5min":6.859,"15min":6.149},"max":{"1min":191.78700000000001,"5min":191.78700000000001,"15min":191.78700000000001},"last":16.759},{"interface":"front","average":{"1min":17.850999999999999,"5min":15.477,"15min":22.440999999999999},"min":{"1min":7.3769999999999998,"5min":7.3769999999999998,"15min":7.0270000000000001},"max":{"1min":111.208,"5min":111.208,"15min":311.13299999999998},"last":13.367000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":22.614999999999998,"5min":17.094999999999999,"15min":18.754999999999999},"min":{"1min":11.262,"5min":10.199,"15min":7.0960000000000001},"max":{"1min":94.787999999999997,"5min":94.787999999999997,"15min":99.623000000000005},"last":33.271000000000001},{"interface":"front","average":{"1min":26.064,"5min":17.611000000000001,"15min":21.722999999999999},"min":{"1min":8.6600000000000001,"5min":8.6080000000000005,"15min":7.9909999999999997},"max":{"1min":153.83799999999999,"5min":153.83799999999999,"15min":186.11000000000001},"last":31.146000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":35.716000000000001,"5min":20.347000000000001,"15min":24.768999999999998},"min":{"1min":12.414999999999999,"5min":9.4819999999999993,"15min":9.1150000000000002},"max":{"1min":207.64599999999999,"5min":207.64599999999999,"15min":207.64599999999999},"last":35.598999999999997},{"interface":"front","average":{"1min":23.535,"5min":17.969999999999999,"15min":21.334},"min":{"1min":11.978999999999999,"5min":9.4469999999999992,"15min":9.0069999999999997},"max":{"1min":74.777000000000001,"5min":74.777000000000001,"15min":148.89599999999999},"last":14.211}]}]},{"osd":3,"up_from":16,"seq":68719477125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70992,"kb_used_data":7228,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300848,"statfs":{"total":96636764160,"available":96564068352,"internally_reserved":0,"allocated":7401472,"data_stored":4081800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":38.427,"5min":15.26,"15min":18.015000000000001},"min":{"1min":6.8440000000000003,"5min":5.5540000000000003,"15min":5.2539999999999996},"max":{"1min":180.16,"5min":180.16,"15min":181.94200000000001},"last":10.919},{"interface":"front","average":{"1min":28.507000000000001,"5min":13.943,"15min":18.838000000000001},"min":{"1min":4.915,"5min":4.915,"15min":4.915},"max":{"1min":107.5,"5min":107.5,"15min":427.36500000000001},"last":13.970000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":30.577999999999999,"5min":15.76,"15min":20.832999999999998},"min":{"1min":8.1419999999999995,"5min":7.585,"15min":6.4039999999999999},"max":{"1min":146.48099999999999,"5min":146.48099999999999,"15min":227.309},"last":7.3890000000000002},{"interface":"front","average":{"1min":33.158000000000001,"5min":15.33,"15min":19.213999999999999},"min":{"1min":8.2870000000000008,"5min":6.9850000000000003,"15min":5.827},"max":{"1min":223.179,"5min":223.179,"15min":421.40600000000001},"last":9.0570000000000004}]},{"osd":2,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":20.815999999999999,"5min":13.278,"15min":22.062000000000001},"min":{"1min":7.9829999999999997,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":86.969999999999999,"5min":86.969999999999999,"15min":451.97899999999998},"last":16.805},{"interface":"front","average":{"1min":29.763999999999999,"5min":15.894,"15min":23.613},"min":{"1min":6.8540000000000001,"5min":6.7190000000000003,"15min":6.5170000000000003},"max":{"1min":223.84,"5min":223.84,"15min":384.58699999999999},"last":14.712999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":33.475999999999999,"5min":17.899999999999999,"15min":21.863},"min":{"1min":9.6440000000000001,"5min":8.5640000000000001,"15min":6.5330000000000004},"max":{"1min":244.989,"5min":244.989,"15min":424.09500000000003},"last":20.486000000000001},{"interface":"front","average":{"1min":31.292000000000002,"5min":17.065000000000001,"15min":21.056000000000001},"min":{"1min":8.9700000000000006,"5min":7.9870000000000001,"15min":7.0049999999999999},"max":{"1min":159.78200000000001,"5min":159.78200000000001,"15min":202.70500000000001},"last":11.794}]},{"osd":5,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":32.497,"5min":19.154,"15min":25.114000000000001},"min":{"1min":10.481999999999999,"5min":9.7400000000000002,"15min":9.0860000000000003},"max":{"1min":228.96700000000001,"5min":228.96700000000001,"15min":424.80099999999999},"last":19.100999999999999},{"interface":"front","average":{"1min":23.297999999999998,"5min":15.457000000000001,"15min":21.076000000000001},"min":{"1min":8.0359999999999996,"5min":7.2850000000000001,"15min":7.1710000000000003},"max":{"1min":87.635000000000005,"5min":87.635000000000005,"15min":231.65199999999999},"last":18.146000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":27.949000000000002,"5min":17.577000000000002,"15min":21.687999999999999},"min":{"1min":8.2859999999999996,"5min":8.2859999999999996,"15min":7.9340000000000002},"max":{"1min":136.684,"5min":136.684,"15min":203.99100000000001},"last":10.311999999999999},{"interface":"front","average":{"1min":30.741,"5min":19.934000000000001,"15min":25.875},"min":{"1min":12.468999999999999,"5min":10.978,"15min":10.978},"max":{"1min":147.88200000000001,"5min":147.88200000000001,"15min":203.34100000000001},"last":13.345000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":25.036000000000001,"5min":16.992999999999999,"15min":22.780000000000001},"min":{"1min":8.7509999999999994,"5min":7.9589999999999996,"15min":7.0259999999999998},"max":{"1min":105.342,"5min":105.342,"15min":196.05699999999999},"last":22.114999999999998},{"interface":"front","average":{"1min":27.623000000000001,"5min":19.024999999999999,"15min":25.555},"min":{"1min":11.186,"5min":11.186,"15min":10.492000000000001},"max":{"1min":102.29000000000001,"5min":102.29000000000001,"15min":236.84700000000001},"last":23.364000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477123,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70996,"kb_used_data":7232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300844,"statfs":{"total":96636764160,"available":96564064256,"internally_reserved":0,"allocated":7405568,"data_stored":4081805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.219000000000001,"5min":14.260999999999999,"15min":20.582999999999998},"min":{"1min":6.4989999999999997,"5min":4.7110000000000003,"15min":4.6059999999999999},"max":{"1min":169.971,"5min":169.971,"15min":231.76499999999999},"last":9.3409999999999993},{"interface":"front","average":{"1min":29.649999999999999,"5min":14.412000000000001,"15min":19.628},"min":{"1min":5.3049999999999997,"5min":5.1890000000000001,"15min":4.8810000000000002},"max":{"1min":150.74199999999999,"5min":150.74199999999999,"15min":222.62},"last":10.647}]},{"osd":1,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":23.131,"5min":12.741,"15min":20.030000000000001},"min":{"1min":6.4980000000000002,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":182.83600000000001,"5min":182.83600000000001,"15min":220.96700000000001},"last":8.6430000000000007},{"interface":"front","average":{"1min":27.039999999999999,"5min":16.943999999999999,"15min":23.343},"min":{"1min":8.6869999999999994,"5min":8.6869999999999994,"15min":7.8639999999999999},"max":{"1min":196.886,"5min":196.886,"15min":213.27799999999999},"last":12.291}]},{"osd":3,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":40.603999999999999,"5min":17.788,"15min":22.448},"min":{"1min":7.0609999999999999,"5min":7.0609999999999999,"15min":6.734},"max":{"1min":177.04900000000001,"5min":177.04900000000001,"15min":219.51300000000001},"last":13.166},{"interface":"front","average":{"1min":41.942999999999998,"5min":19.474,"15min":24.347000000000001},"min":{"1min":7.6740000000000004,"5min":7.6740000000000004,"15min":6.9900000000000002},"max":{"1min":204.721,"5min":204.721,"15min":204.721},"last":19.106999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.542000000000002,"5min":16.068000000000001,"15min":21.274000000000001},"min":{"1min":8.6310000000000002,"5min":7.2060000000000004,"15min":6.6799999999999997},"max":{"1min":180.369,"5min":180.369,"15min":251.745},"last":26.879000000000001},{"interface":"front","average":{"1min":33.243000000000002,"5min":18.972000000000001,"15min":24.882999999999999},"min":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":7.5720000000000001},"max":{"1min":170.637,"5min":170.637,"15min":287.32100000000003},"last":25.776}]},{"osd":5,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":37.953000000000003,"5min":18.436,"15min":24.466999999999999},"min":{"1min":8.5060000000000002,"5min":8.3300000000000001,"15min":8.0020000000000007},"max":{"1min":206.56700000000001,"5min":206.56700000000001,"15min":210.012},"last":11.662000000000001},{"interface":"front","average":{"1min":33.348999999999997,"5min":19.402000000000001,"15min":25.024000000000001},"min":{"1min":9.1980000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"max":{"1min":232.91900000000001,"5min":232.91900000000001,"15min":232.91900000000001},"last":14.550000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":35.573999999999998,"5min":19.463000000000001,"15min":23.882999999999999},"min":{"1min":8.8710000000000004,"5min":8.8710000000000004,"15min":8.1829999999999998},"max":{"1min":182.07400000000001,"5min":182.07400000000001,"15min":237.21899999999999},"last":17.206},{"interface":"front","average":{"1min":38.033999999999999,"5min":20.099,"15min":27.759},"min":{"1min":12.598000000000001,"5min":9.5960000000000001,"15min":9.5649999999999995},"max":{"1min":185.90299999999999,"5min":185.90299999999999,"15min":241.92699999999999},"last":17.835000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":19.201000000000001,"5min":16.905000000000001,"15min":24.222000000000001},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.5690000000000008},"max":{"1min":68.075000000000003,"5min":68.075000000000003,"15min":341.73000000000002},"last":15.349},{"interface":"front","average":{"1min":22.43,"5min":18.937999999999999,"15min":27.407},"min":{"1min":10.853,"5min":10.853,"15min":10.853},"max":{"1min":89.631,"5min":89.631,"15min":343.17399999999998},"last":16.285}]}]},{"osd":0,"up_from":15,"seq":64424509828,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70588,"kb_used_data":7252,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301252,"statfs":{"total":96636764160,"available":96564482048,"internally_reserved":0,"allocated":7426048,"data_stored":4096103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 17:36:15 2023","interfaces":[{"interface":"back","average":{"1min":11.167,"5min":10.766999999999999,"15min":17.079000000000001},"min":{"1min":5.5579999999999998,"5min":5.5579999999999998,"15min":4.3550000000000004},"max":{"1min":33.322000000000003,"5min":33.322000000000003,"15min":210.43299999999999},"last":66.680999999999997},{"interface":"front","average":{"1min":12.661,"5min":10.788,"15min":18.704999999999998},"min":{"1min":6.242,"5min":6.0750000000000002,"15min":5.2590000000000003},"max":{"1min":44.020000000000003,"5min":44.020000000000003,"15min":257.30500000000001},"last":92.177000000000007}]},{"osd":2,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.808,"5min":14.16,"15min":19.648},"min":{"1min":8.2010000000000005,"5min":7.4740000000000002,"15min":6.452},"max":{"1min":94.155000000000001,"5min":94.155000000000001,"15min":239.24600000000001},"last":94.155000000000001},{"interface":"front","average":{"1min":16.765000000000001,"5min":13.634,"15min":18.114999999999998},"min":{"1min":8.3179999999999996,"5min":6.8579999999999997,"15min":5.6520000000000001},"max":{"1min":86.099999999999994,"5min":86.099999999999994,"15min":246.28299999999999},"last":86.099999999999994}]},{"osd":3,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":12.295999999999999,"15min":19.577000000000002},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"max":{"1min":128.08099999999999,"5min":128.08099999999999,"15min":240.017},"last":53.909999999999997},{"interface":"front","average":{"1min":28.396000000000001,"5min":17.071999999999999,"15min":25.015999999999998},"min":{"1min":9.452,"5min":8.3499999999999996,"15min":7.2779999999999996},"max":{"1min":136.40600000000001,"5min":136.40600000000001,"15min":249.86099999999999},"last":93.147000000000006}]},{"osd":4,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":15.007999999999999,"5min":12.295,"15min":18.792999999999999},"min":{"1min":7.5780000000000003,"5min":6.6909999999999998,"15min":6.6909999999999998},"max":{"1min":77.429000000000002,"5min":77.429000000000002,"15min":241.292},"last":196.71799999999999},{"interface":"front","average":{"1min":19.893999999999998,"5min":16.827000000000002,"15min":22.696000000000002},"min":{"1min":11.875,"5min":9.3759999999999994,"15min":7.2290000000000001},"max":{"1min":91.792000000000002,"5min":91.792000000000002,"15min":236.65799999999999},"last":207.38}]},{"osd":5,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":16.934000000000001,"5min":14.622,"15min":24.321999999999999},"min":{"1min":9.2949999999999999,"5min":8.6929999999999996,"15min":7.9400000000000004},"max":{"1min":45.228000000000002,"5min":45.228000000000002,"15min":241.91200000000001},"last":55.027000000000001},{"interface":"front","average":{"1min":16.408000000000001,"5min":14.084,"15min":21.462},"min":{"1min":7.2779999999999996,"5min":7.2779999999999996,"15min":6.4279999999999999},"max":{"1min":66.477000000000004,"5min":66.477000000000004,"15min":247.185},"last":73.453000000000003}]},{"osd":6,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":15.752000000000001,"15min":23.873999999999999},"min":{"1min":9.8810000000000002,"5min":8.2829999999999995,"15min":7.923},"max":{"1min":50.194000000000003,"5min":50.194000000000003,"15min":244.93799999999999},"last":68.665999999999997},{"interface":"front","average":{"1min":16.581,"5min":15.106999999999999,"15min":22.677},"min":{"1min":9.2609999999999992,"5min":9.2110000000000003,"15min":7.734},"max":{"1min":42.387999999999998,"5min":42.387999999999998,"15min":247.80500000000001},"last":83.817999999999998}]},{"osd":7,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":18.469999999999999,"5min":17.091999999999999,"15min":24.052},"min":{"1min":11.074999999999999,"5min":10.281000000000001,"15min":9.0820000000000007},"max":{"1min":37.758000000000003,"5min":37.758000000000003,"15min":238.58099999999999},"last":199.01400000000001},{"interface":"front","average":{"1min":18.225000000000001,"5min":16.898,"15min":24.481000000000002},"min":{"1min":11.706,"5min":9.3140000000000001,"15min":9.3140000000000001},"max":{"1min":36.850000000000001,"5min":36.850000000000001,"15min":273.142},"last":91.361000000000004}]}]},{"osd":6,"up_from":14,"seq":60129542534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70568,"kb_used_data":7232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301272,"statfs":{"total":96636764160,"available":96564502528,"internally_reserved":0,"allocated":7405568,"data_stored":4081805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":10.34,"5min":9.7390000000000008,"15min":13.387},"min":{"1min":4.8339999999999996,"5min":4.577,"15min":4.0730000000000004},"max":{"1min":25.731999999999999,"5min":25.731999999999999,"15min":129.89699999999999},"last":6.7709999999999999},{"interface":"front","average":{"1min":11.954000000000001,"5min":10.832000000000001,"15min":15.518000000000001},"min":{"1min":6.5010000000000003,"5min":6.4669999999999996,"15min":4.8129999999999997},"max":{"1min":34.526000000000003,"5min":34.526000000000003,"15min":155.99100000000001},"last":12.582000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":14.457000000000001,"5min":12.138,"15min":15.375},"min":{"1min":7.9969999999999999,"5min":6.702,"15min":6.2789999999999999},"max":{"1min":50.130000000000003,"5min":50.130000000000003,"15min":175.55000000000001},"last":10.417},{"interface":"front","average":{"1min":18.449000000000002,"5min":15.638999999999999,"15min":19.158000000000001},"min":{"1min":10.378,"5min":10.378,"15min":7.0490000000000004},"max":{"1min":54.060000000000002,"5min":54.060000000000002,"15min":130.91999999999999},"last":11.733000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":21.776,"5min":15.327,"15min":19.893999999999998},"min":{"1min":7.0010000000000003,"5min":6.8879999999999999,"15min":6.8879999999999999},"max":{"1min":172.459,"5min":172.459,"15min":191.71299999999999},"last":13.83},{"interface":"front","average":{"1min":19.948,"5min":11.276999999999999,"15min":16.713000000000001},"min":{"1min":4.4370000000000003,"5min":4.4370000000000003,"15min":4.4370000000000003},"max":{"1min":174.608,"5min":174.608,"15min":244.46299999999999},"last":5.1449999999999996}]},{"osd":3,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":19.504999999999999,"5min":15.193,"15min":20.762},"min":{"1min":10.653,"5min":8.2929999999999993,"15min":6.5830000000000002},"max":{"1min":70.066999999999993,"5min":70.066999999999993,"15min":223.97800000000001},"last":16.021999999999998},{"interface":"front","average":{"1min":18.291,"5min":14.847,"15min":22.268000000000001},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":6.8559999999999999},"max":{"1min":71.659999999999997,"5min":71.659999999999997,"15min":212.309},"last":17.539999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.582000000000001,"5min":14.68,"15min":19.289999999999999},"min":{"1min":10.567,"5min":8.952,"15min":7.0510000000000002},"max":{"1min":21.972000000000001,"5min":22.463000000000001,"15min":134.95699999999999},"last":15.204000000000001},{"interface":"front","average":{"1min":17.640999999999998,"5min":17.771000000000001,"15min":22.725000000000001},"min":{"1min":11.27,"5min":11.27,"15min":9.9809999999999999},"max":{"1min":22.663,"5min":22.663,"15min":135.798},"last":19.507000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.782,"5min":15.356,"15min":20.988},"min":{"1min":7.8040000000000003,"5min":7.8040000000000003,"15min":7.2869999999999999},"max":{"1min":20.332000000000001,"5min":20.966000000000001,"15min":288.577},"last":13.218999999999999},{"interface":"front","average":{"1min":17.925999999999998,"5min":17.777000000000001,"15min":23.446999999999999},"min":{"1min":12.861000000000001,"5min":12.124000000000001,"15min":10.234},"max":{"1min":23.948,"5min":23.948,"15min":142.77699999999999},"last":18.559999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":11.468999999999999,"5min":10.596,"15min":15.210000000000001},"min":{"1min":7.2649999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"max":{"1min":21.209,"5min":21.209,"15min":168.59700000000001},"last":9.2189999999999994},{"interface":"front","average":{"1min":17.047000000000001,"5min":16.718,"15min":21.393000000000001},"min":{"1min":11.954000000000001,"5min":8.9459999999999997,"15min":8.4770000000000003},"max":{"1min":24.709,"5min":25.518000000000001,"15min":183.054},"last":16.818000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542534,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75748,"kb_used_data":8032,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296092,"statfs":{"total":96636764160,"available":96559198208,"internally_reserved":0,"allocated":8224768,"data_stored":4735673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":58,"apply_latency_ms":58,"commit_latency_ns":58000000,"apply_latency_ns":58000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":24.530999999999999,"5min":11.712,"15min":15.721},"min":{"1min":4.3620000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"max":{"1min":201.333,"5min":201.333,"15min":228.261},"last":73.542000000000002},{"interface":"front","average":{"1min":27.792000000000002,"5min":13.209,"15min":17.690999999999999},"min":{"1min":6.5709999999999997,"5min":5.0339999999999998,"15min":4.6580000000000004},"max":{"1min":221.876,"5min":221.876,"15min":222.441},"last":85.289000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.556000000000001,"5min":12.199999999999999,"15min":19.036000000000001},"min":{"1min":7.0490000000000004,"5min":6.0640000000000001,"15min":4.8899999999999997},"max":{"1min":66.040999999999997,"5min":66.040999999999997,"15min":216.20099999999999},"last":72.462999999999994},{"interface":"front","average":{"1min":17.574000000000002,"5min":12.794,"15min":19.777000000000001},"min":{"1min":7.7430000000000003,"5min":6.5650000000000004,"15min":5.6210000000000004},"max":{"1min":65.156999999999996,"5min":65.156999999999996,"15min":255.95599999999999},"last":28.507999999999999}]},{"osd":2,"last update":"Fri Dec 1 17:35:58 2023","interfaces":[{"interface":"back","average":{"1min":12.641999999999999,"5min":11.792,"15min":17.571999999999999},"min":{"1min":6.468,"5min":6.468,"15min":6.468},"max":{"1min":39.947000000000003,"5min":39.947000000000003,"15min":179.81999999999999},"last":129.893},{"interface":"front","average":{"1min":16.623999999999999,"5min":15.505000000000001,"15min":22.356999999999999},"min":{"1min":8.7070000000000007,"5min":8.7070000000000007,"15min":8.7070000000000007},"max":{"1min":42.759,"5min":42.759,"15min":224.47300000000001},"last":130.87200000000001}]},{"osd":3,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":13.112,"15min":19.452999999999999},"min":{"1min":6.819,"5min":6.819,"15min":5.9020000000000001},"max":{"1min":61.933,"5min":61.933,"15min":248.07300000000001},"last":7.1609999999999996},{"interface":"front","average":{"1min":22.122,"5min":16.693000000000001,"15min":23.396000000000001},"min":{"1min":11.053000000000001,"5min":10.221,"15min":9.7989999999999995},"max":{"1min":62.886000000000003,"5min":62.886000000000003,"15min":235.02699999999999},"last":8.9320000000000004}]},{"osd":4,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":17.068999999999999,"5min":15.241,"15min":19.116},"min":{"1min":10.965999999999999,"5min":9.4930000000000003,"15min":8.1699999999999999},"max":{"1min":59.747,"5min":59.747,"15min":211.09399999999999},"last":16.664000000000001},{"interface":"front","average":{"1min":14.052,"5min":14.444000000000001,"15min":18.030000000000001},"min":{"1min":9.6010000000000009,"5min":8.6850000000000005,"15min":7.0540000000000003},"max":{"1min":29.402000000000001,"5min":29.402000000000001,"15min":216.85300000000001},"last":13.747}]},{"osd":6,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":20.969999999999999,"5min":17.178000000000001,"15min":21.32},"min":{"1min":11.702,"5min":10.103,"15min":8.4339999999999993},"max":{"1min":111.396,"5min":111.396,"15min":223.89500000000001},"last":12.025},{"interface":"front","average":{"1min":16.765999999999998,"5min":14.01,"15min":14.154999999999999},"min":{"1min":7.5949999999999998,"5min":5.6829999999999998,"15min":5.6479999999999997},"max":{"1min":34.613,"5min":34.613,"15min":159.84399999999999},"last":8.0660000000000007}]},{"osd":7,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":18.885999999999999,"5min":17.652000000000001,"15min":21.867000000000001},"min":{"1min":10.467000000000001,"5min":10.467000000000001,"15min":9.6430000000000007},"max":{"1min":62.256999999999998,"5min":62.256999999999998,"15min":226.06100000000001},"last":10.708},{"interface":"front","average":{"1min":21.974,"5min":17.885000000000002,"15min":21.588999999999999},"min":{"1min":9.3369999999999997,"5min":9.3369999999999997,"15min":8.1820000000000004},"max":{"1min":125.431,"5min":125.431,"15min":317.267},"last":12.648999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542535,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75536,"kb_used_data":7636,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296304,"statfs":{"total":96636764160,"available":96559415296,"internally_reserved":0,"allocated":7819264,"data_stored":4337825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":7.8300000000000001,"5min":9.3949999999999996,"15min":16.763000000000002},"min":{"1min":4.1109999999999998,"5min":4.1109999999999998,"15min":3.7690000000000001},"max":{"1min":15.778,"5min":24.963999999999999,"15min":521.55899999999997},"last":4.609},{"interface":"front","average":{"1min":9.3529999999999998,"5min":9.8160000000000007,"15min":15.084},"min":{"1min":6.3360000000000003,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":13.864000000000001,"5min":21.491,"15min":281.125},"last":9.0220000000000002}]},{"osd":1,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":11.353999999999999,"5min":11.315,"15min":19.023},"min":{"1min":8.2140000000000004,"5min":6.7969999999999997,"15min":6.2770000000000001},"max":{"1min":14.935,"5min":21.539000000000001,"15min":308.99299999999999},"last":10.351000000000001},{"interface":"front","average":{"1min":16.753,"5min":15.666,"15min":22.716000000000001},"min":{"1min":10.832000000000001,"5min":9.1069999999999993,"15min":6.3079999999999998},"max":{"1min":31.140000000000001,"5min":31.140000000000001,"15min":289.79399999999998},"last":11.068}]},{"osd":2,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":16.849,"5min":15.061999999999999,"15min":21.503},"min":{"1min":11.24,"5min":9.1029999999999998,"15min":5.976},"max":{"1min":27.797999999999998,"5min":27.797999999999998,"15min":298.18200000000002},"last":14.058999999999999},{"interface":"front","average":{"1min":12.192,"5min":10.987,"15min":16.457999999999998},"min":{"1min":8.0640000000000001,"5min":6.4820000000000002,"15min":6.4820000000000002},"max":{"1min":25.771999999999998,"5min":25.771999999999998,"15min":223.768},"last":11.661}]},{"osd":3,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":12.680999999999999,"15min":18.734000000000002},"min":{"1min":8.6850000000000005,"5min":7.1580000000000004,"15min":6.2169999999999996},"max":{"1min":26.462,"5min":26.462,"15min":243.40299999999999},"last":15.862},{"interface":"front","average":{"1min":17.468,"5min":16.356999999999999,"15min":22.004000000000001},"min":{"1min":13.864000000000001,"5min":10.156000000000001,"15min":8.4390000000000001},"max":{"1min":37.262999999999998,"5min":37.262999999999998,"15min":253.56100000000001},"last":17.404}]},{"osd":5,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":17.428999999999998,"5min":15.695,"15min":22.879999999999999},"min":{"1min":10.804,"5min":9.5199999999999996,"15min":6.6349999999999998},"max":{"1min":35.213999999999999,"5min":35.213999999999999,"15min":336.55599999999998},"last":16.579000000000001},{"interface":"front","average":{"1min":10.366,"5min":11.09,"15min":19.364000000000001},"min":{"1min":7.0019999999999998,"5min":6.4859999999999998,"15min":6.4790000000000001},"max":{"1min":14.853,"5min":23.786000000000001,"15min":357.50900000000001},"last":8.2149999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":14.206,"5min":15.972,"15min":20.510999999999999},"min":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":8.1080000000000005},"max":{"1min":34.234000000000002,"5min":34.234000000000002,"15min":255.52500000000001},"last":9.6910000000000007},{"interface":"front","average":{"1min":19.222000000000001,"5min":15.115,"15min":19.218},"min":{"1min":15.247999999999999,"5min":7.7080000000000002,"15min":7.7080000000000002},"max":{"1min":37.884,"5min":37.884,"15min":312.791},"last":18.312000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":15.161,"5min":15.696,"15min":23.895},"min":{"1min":9.7590000000000003,"5min":9.6940000000000008,"15min":8.7330000000000005},"max":{"1min":23.195,"5min":23.195,"15min":413.48899999999998},"last":13.362},{"interface":"front","average":{"1min":17.57,"5min":16.561,"15min":21.308},"min":{"1min":9.4770000000000003,"5min":9.0809999999999995,"15min":6.6820000000000004},"max":{"1min":30.414999999999999,"5min":30.414999999999999,"15min":417.952},"last":15.034000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575241,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70972,"kb_used_data":7432,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300868,"statfs":{"total":96636764160,"available":96564088832,"internally_reserved":0,"allocated":7610368,"data_stored":4206728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":13.364000000000001,"5min":11.880000000000001,"15min":16.117000000000001},"min":{"1min":8.8580000000000005,"5min":5.2729999999999997,"15min":4.3360000000000003},"max":{"1min":39.423000000000002,"5min":39.423000000000002,"15min":203.19300000000001},"last":8.9830000000000005},{"interface":"front","average":{"1min":17.373999999999999,"5min":15.071999999999999,"15min":18.241},"min":{"1min":11.513,"5min":6.4269999999999996,"15min":4.4199999999999999},"max":{"1min":40.698999999999998,"5min":40.698999999999998,"15min":204.13},"last":14.613}]},{"osd":1,"last update":"Fri Dec 1 17:35:57 2023","interfaces":[{"interface":"back","average":{"1min":7.9080000000000004,"5min":8.2289999999999992,"15min":15.226000000000001},"min":{"1min":6.1139999999999999,"5min":5.2300000000000004,"15min":5.0369999999999999},"max":{"1min":10.85,"5min":17.564,"15min":167.82900000000001},"last":6.9969999999999999},{"interface":"front","average":{"1min":15.026,"5min":14.375999999999999,"15min":16.369},"min":{"1min":11.634,"5min":7.4870000000000001,"15min":5.484},"max":{"1min":22.280999999999999,"5min":22.280999999999999,"15min":154.137},"last":12.805}]},{"osd":2,"last update":"Fri Dec 1 17:36:23 2023","interfaces":[{"interface":"back","average":{"1min":11.753,"5min":10.013999999999999,"15min":16.608000000000001},"min":{"1min":5.8700000000000001,"5min":5.1820000000000004,"15min":5.1820000000000004},"max":{"1min":26.274999999999999,"5min":26.274999999999999,"15min":230.40600000000001},"last":13.621},{"interface":"front","average":{"1min":14.275,"5min":12.394,"15min":17.012},"min":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"max":{"1min":29.861000000000001,"5min":29.861000000000001,"15min":268.68000000000001},"last":9.5510000000000002}]},{"osd":3,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":13.083,"15min":20.297000000000001},"min":{"1min":8.1359999999999992,"5min":6.5110000000000001,"15min":6.5110000000000001},"max":{"1min":85.724000000000004,"5min":85.724000000000004,"15min":290.15300000000002},"last":8.1999999999999993},{"interface":"front","average":{"1min":14.601000000000001,"5min":11.683,"15min":20.149000000000001},"min":{"1min":6.6319999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"max":{"1min":83.183000000000007,"5min":83.183000000000007,"15min":305.41399999999999},"last":6.1669999999999998}]},{"osd":4,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":14.577,"5min":14.051,"15min":18.489000000000001},"min":{"1min":10.007999999999999,"5min":8.5489999999999995,"15min":6.9299999999999997},"max":{"1min":17.609999999999999,"5min":22.071000000000002,"15min":149.93000000000001},"last":11.746},{"interface":"front","average":{"1min":17.727,"5min":17.503,"15min":23.488},"min":{"1min":14.632,"5min":11.496,"15min":9.7810000000000006},"max":{"1min":26.896999999999998,"5min":27.422000000000001,"15min":239.37},"last":16.341000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":17.140000000000001,"5min":13.266,"15min":20.885000000000002},"min":{"1min":10.646000000000001,"5min":7.3209999999999997,"15min":6.8460000000000001},"max":{"1min":26.053999999999998,"5min":26.053999999999998,"15min":282.31200000000001},"last":17.155000000000001},{"interface":"front","average":{"1min":9.9749999999999996,"5min":13.909000000000001,"15min":19.908999999999999},"min":{"1min":6.8929999999999998,"5min":6.1529999999999996,"15min":6.1529999999999996},"max":{"1min":17.657,"5min":20.282,"15min":193.83600000000001},"last":11.114000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":18.216999999999999,"5min":17.367999999999999,"15min":20.030999999999999},"min":{"1min":13.897,"5min":11.176,"15min":6.625},"max":{"1min":24.832999999999998,"5min":24.832999999999998,"15min":209.25},"last":15.367000000000001},{"interface":"front","average":{"1min":19.565999999999999,"5min":19.302,"15min":24.709},"min":{"1min":15.249000000000001,"5min":14.266999999999999,"15min":8.9420000000000002},"max":{"1min":27.905000000000001,"5min":27.905000000000001,"15min":193.184},"last":17.757000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:36:28.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:36:28.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.509+0000 7fa571632640 10 monclient: get_monmap_and_config 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.509+0000 7fa571632640 10 monclient: build_initial_monmap 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.509+0000 7fa571632640 10 monclient: monmap: 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:28.510100+0000 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:28.510100+0000 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:28.509 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:28.510 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:28.510 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.509+0000 7fa571632640 1 Processor -- start 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 1 -- start start 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient: init 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa56c0af380 0x7fa56c06fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): picked mon.noname-a con 0x7fa56c0af380 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa56c072090 0x7fa56c0724e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c072090 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c13b4c0 0x7fa56c13b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:28.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): picked mon.noname-b con 0x7fa56c13b4c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c0af760 con 0x7fa56c0af380 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c0702b0 con 0x7fa56c072090 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c070590 con 0x7fa56c13b4c0 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.510+0000 7fa571632640 10 monclient(hunting): authenticate will time out at 3319.748124s 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56a7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa56c072090 0x7fa56c0724e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56b7fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c13b4c0 0x7fa56c13b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56affd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa56c0af380 0x7fa56c06fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56a7fc640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fa56c072090 0x7fa56c0724e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47678/0 (socket says 172.21.15.67:47678) 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56a7fc640 1 -- 172.21.15.67:0/1119499985 learned_addr learned my addr 172.21.15.67:0/1119499985 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request con 0x7fa56c072090 auth_method 0 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:28.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56affd640 10 monclient(hunting): get_auth_request con 0x7fa56c0af380 auth_method 0 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request con 0x7fa56c13b4c0 auth_method 0 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.511+0000 7fa56b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:28.514 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.512+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:28.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_done global_id 23137 payload 274 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 1 -- 172.21.15.67:0/1119499985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c13b4c0 msgr2=0x7fa56c13b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 1 --2- 172.21.15.67:0/1119499985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c13b4c0 0x7fa56c13b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 1 -- 172.21.15.67:0/1119499985 >> v2:172.21.15.67:3300/0 conn(0x7fa56c0af380 msgr2=0x7fa56c06fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 1 --2- 172.21.15.67:0/1119499985 >> v2:172.21.15.67:3300/0 conn(0x7fa56c0af380 0x7fa56c06fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 10 monclient: _finish_hunting 0 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 1 monclient: found mon.noname-c 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.513+0000 7fa56a7fc640 1 -- 172.21.15.67:0/1119499985 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c0add10 con 0x7fa56c072090 2023-12-01T17:36:28.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.515+0000 7fa56a7fc640 1 --2- 172.21.15.67:0/1119499985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c0724e0 secure :-1 s=READY pgs=5878 cs=0 l=1 rev1=1 crypto rx=0x7fa56c13e380 tx=0x7fa56002b7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb7157b5aea84a12 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 1 -- 172.21.15.67:0/1119499985 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560031750 con 0x7fa56c072090 2023-12-01T17:36:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: dump: 2023-12-01T17:36:28.518 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:28.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 1 -- 172.21.15.67:0/1119499985 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5600318b0 con 0x7fa56c072090 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 5 monclient: authenticate success, global_id 23137 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 1 -- 172.21.15.67:0/1119499985 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa560031c40 con 0x7fa56c072090 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 4 set_mon_vals no callback set 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa569ffb640 10 monclient: dump: 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.520 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:28.521 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:28.521 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 10 monclient: shutdown 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 -- 172.21.15.67:0/1119499985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 msgr2=0x7fa56c0724e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 --2- 172.21.15.67:0/1119499985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c0724e0 secure :-1 s=READY pgs=5878 cs=0 l=1 rev1=1 crypto rx=0x7fa56c13e380 tx=0x7fa56002b7f0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 -- 172.21.15.67:0/1119499985 shutdown_connections 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 --2- 172.21.15.67:0/1119499985 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c13b4c0 0x7fa56c13b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 --2- 172.21.15.67:0/1119499985 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c0724e0 unknown :-1 s=CLOSED pgs=5878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 --2- 172.21.15.67:0/1119499985 >> v2:172.21.15.67:3300/0 conn(0x7fa56c0af380 0x7fa56c06fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 -- 172.21.15.67:0/1119499985 >> 172.21.15.67:0/1119499985 conn(0x7fa56c0b3ef0 msgr2=0x7fa56c0b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 -- 172.21.15.67:0/1119499985 shutdown_connections 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.519+0000 7fa571632640 1 -- 172.21.15.67:0/1119499985 wait complete. 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.520+0000 7fa571632640 10 monclient: build_initial_monmap 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.520+0000 7fa571632640 10 monclient: monmap: 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:28.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.520+0000 7fa571632640 1 librados: starting msgr at 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.520+0000 7fa571632640 1 librados: starting objecter 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 Processor -- start 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 -- start start 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 librados: setting wanted keys 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 librados: calling monclient init 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient: init 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c1371c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): picked mon.noname-b con 0x7fa56c072090 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c0af380 0x7fa56c137700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c0af380 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa56c13b4c0 0x7fa56c0b1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): picked mon.noname-a con 0x7fa56c13b4c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:28.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c138680 con 0x7fa56c13b4c0 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c138960 con 0x7fa56c072090 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c1c44f0 con 0x7fa56c0af380 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.521+0000 7fa571632640 10 monclient(hunting): authenticate will time out at 3319.759273s 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c1371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56a7fc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c0af380 0x7fa56c137700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa56c13b4c0 0x7fa56c0b1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56affd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c1371c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47680/0 (socket says 172.21.15.67:47680) 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56affd640 1 -- 172.21.15.67:0/2807204895 learned_addr learned my addr 172.21.15.67:0/2807204895 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56affd640 10 monclient(hunting): get_auth_request con 0x7fa56c072090 auth_method 0 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56affd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:28.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56affd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request con 0x7fa56c13b4c0 auth_method 0 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.522+0000 7fa56b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request con 0x7fa56c0af380 auth_method 0 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56a7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56a7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:28.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.523+0000 7fa56b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 10 monclient(hunting): handle_auth_done global_id 23140 payload 824 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 1 -- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c0af380 msgr2=0x7fa56c137700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c0af380 0x7fa56c137700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 1 -- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa56c13b4c0 msgr2=0x7fa56c0b1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa56c13b4c0 0x7fa56c0b1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 10 monclient: _finish_hunting 0 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 1 monclient: found mon.noname-b 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:28.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.525+0000 7fa56affd640 1 -- 172.21.15.67:0/2807204895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5600094d0 con 0x7fa56c072090 2023-12-01T17:36:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.527+0000 7fa56affd640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c1371c0 secure :-1 s=READY pgs=5879 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009e30 tx=0x7fa55c0075c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.527+0000 7fa5537fe640 10 client.?.objecter ms_handle_connect 0x7fa56c072090 2023-12-01T17:36:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.527+0000 7fa5537fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.530+0000 7fa5537fe640 1 -- 172.21.15.67:0/2807204895 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c011070 con 0x7fa56c072090 2023-12-01T17:36:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.530+0000 7fa5537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.530+0000 7fa5537fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:28.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.530+0000 7fa5537fe640 10 monclient: dump: 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 10 monclient: _finish_auth 0 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 10 monclient: _check_auth_tickets 2023-12-01T17:36:28.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 1 -- 172.21.15.67:0/2807204895 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55c016470 con 0x7fa56c072090 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 1 -- 172.21.15.67:0/2807204895 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00add0 con 0x7fa56c072090 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 5 monclient: authenticate success, global_id 23140 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 10 monclient: _renew_subs 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c13bfd0 con 0x7fa56c072090 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5697fa640 4 set_mon_vals no callback set 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa5537fe640 10 monclient: dump: 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:28.534 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 10 client.23140.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 10 monclient: _renew_subs 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c0b3110 con 0x7fa56c072090 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 1 librados: init done 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.531+0000 7fa571632640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.532+0000 7fa571632640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:28.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.532+0000 7fa571632640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:28.536 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.532+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c06f350 con 0x7fa56c072090 2023-12-01T17:36:28.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.550+0000 7fa5537fe640 1 -- 172.21.15.67:0/2807204895 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fa55c0167e0 con 0x7fa56c072090 2023-12-01T17:36:28.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.550+0000 7fa5537fe640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa55806c820 0x7fa55806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:28.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.550+0000 7fa56a7fc640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa55806c820 0x7fa55806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:28.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.550+0000 7fa56a7fc640 10 monclient: get_auth_request con 0x7fa55806c820 auth_method 0 2023-12-01T17:36:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.551+0000 7fa56a7fc640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa55806c820 0x7fa55806ece0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa560035000 tx=0x7fa560008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.551+0000 7fa5537fe640 10 client.23140.objecter ms_handle_connect 0x7fa55806c820 2023-12-01T17:36:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.551+0000 7fa5537fe640 1 -- 172.21.15.67:0/2807204895 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1006..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fa55c058e10 con 0x7fa56c072090 2023-12-01T17:36:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.551+0000 7fa5537fe640 10 client.23140.objecter ms_dispatch 0x7fa56c000b90 osd_map(1099..1099 src has 1006..1099) v4 2023-12-01T17:36:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.551+0000 7fa5537fe640 3 client.23140.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.551+0000 7fa5537fe640 3 client.23140.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:28.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.551+0000 7fa5537fe640 20 client.23140.objecter dump_active .. 0 homeless 2023-12-01T17:36:28.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.786+0000 7fa5537fe640 1 -- 172.21.15.67:0/2807204895 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa55c055150 con 0x7fa56c072090 2023-12-01T17:36:28.796 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.786+0000 7fa5537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:28.797 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.786+0000 7fa5537fe640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:28.961 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.961+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa56c1c4930 con 0x7fa55806c820 2023-12-01T17:36:28.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.966+0000 7fa5537fe640 1 -- 172.21.15.67:0/2807204895 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151451 (secure 0 0 0) 0x7fa560041070 con 0x7fa55806c820 2023-12-01T17:36:28.967 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:36:28.967 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:36:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.968+0000 7fa571632640 10 librados: watch_flush enter 2023-12-01T17:36:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.968+0000 7fa571632640 10 librados: watch_flush exit 2023-12-01T17:36:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.968+0000 7fa571632640 20 client.23140.objecter shutdown clearing up homeless session... 2023-12-01T17:36:28.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.968+0000 7fa571632640 10 client.23140.objecter successfully canceled tick 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.968+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa55806c820 msgr2=0x7fa55806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.968+0000 7fa571632640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa55806c820 0x7fa55806ece0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa560035000 tx=0x7fa560008030 comp rx=0 tx=0).stop 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 10 monclient: shutdown 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 msgr2=0x7fa56c1371c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c1371c0 secure :-1 s=READY pgs=5879 cs=0 l=1 rev1=1 crypto rx=0x7fa55c009e30 tx=0x7fa55c0075c0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 shutdown_connections 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa56c13b4c0 0x7fa56c0b1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fa55806c820 0x7fa55806ece0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fa56c0af380 0x7fa56c137700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 --2- 172.21.15.67:0/2807204895 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa56c072090 0x7fa56c1371c0 unknown :-1 s=CLOSED pgs=5879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 >> 172.21.15.67:0/2807204895 conn(0x7fa56c0b3ef0 msgr2=0x7fa56c06ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 shutdown_connections 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 -- 172.21.15.67:0/2807204895 wait complete. 2023-12-01T17:36:28.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:28.969+0000 7fa571632640 1 librados: shutdown 2023-12-01T17:36:28.979 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":2081,"stamp":"2023-12-01T17:36:28.773857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":575752,"kb_used_data":59100,"kb_used_omap":12,"kb_used_meta":513139,"kb_avail":754398968,"statfs":{"total":773094113280,"available":772504543232,"internally_reserved":0,"allocated":60518400,"data_stored":33737203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":525454896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":409,"apply_latency_ms":409,"commit_latency_ns":409000000,"apply_latency_ns":409000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000700"},"pg_stats":[{"pgid":"22.6","version":"0'0","reported_seq":1045,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.677997+0000","last_change":"2023-12-01T17:05:14.287690+0000","last_active":"2023-12-01T17:36:22.677997+0000","last_peered":"2023-12-01T17:36:22.677997+0000","last_clean":"2023-12-01T17:36:22.677997+0000","last_became_active":"2023-12-01T17:05:14.257152+0000","last_became_peered":"2023-12-01T17:05:14.257152+0000","last_unstale":"2023-12-01T17:36:22.677997+0000","last_undegraded":"2023-12-01T17:36:22.677997+0000","last_fullsized":"2023-12-01T17:36:22.677997+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:59:48.591916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1053,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.536581+0000","last_change":"2023-12-01T17:05:21.303515+0000","last_active":"2023-12-01T17:36:21.536581+0000","last_peered":"2023-12-01T17:36:21.536581+0000","last_clean":"2023-12-01T17:36:21.536581+0000","last_became_active":"2023-12-01T17:05:21.296791+0000","last_became_peered":"2023-12-01T17:05:21.296791+0000","last_unstale":"2023-12-01T17:36:21.536581+0000","last_undegraded":"2023-12-01T17:36:21.536581+0000","last_fullsized":"2023-12-01T17:36:21.536581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:51:36.558228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1056,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.138780+0000","last_change":"2023-12-01T17:05:13.270729+0000","last_active":"2023-12-01T17:36:26.138780+0000","last_peered":"2023-12-01T17:36:26.138780+0000","last_clean":"2023-12-01T17:36:26.138780+0000","last_became_active":"2023-12-01T17:05:13.265267+0000","last_became_peered":"2023-12-01T17:05:13.265267+0000","last_unstale":"2023-12-01T17:36:26.138780+0000","last_undegraded":"2023-12-01T17:36:26.138780+0000","last_fullsized":"2023-12-01T17:36:26.138780+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:57:02.423320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1049,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.596184+0000","last_change":"2023-12-01T17:05:38.668209+0000","last_active":"2023-12-01T17:36:22.596184+0000","last_peered":"2023-12-01T17:36:22.596184+0000","last_clean":"2023-12-01T17:36:22.596184+0000","last_became_active":"2023-12-01T17:05:38.651540+0000","last_became_peered":"2023-12-01T17:05:38.651540+0000","last_unstale":"2023-12-01T17:36:22.596184+0000","last_undegraded":"2023-12-01T17:36:22.596184+0000","last_fullsized":"2023-12-01T17:36:22.596184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:43:47.235945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1058,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.693835+0000","last_change":"2023-12-01T17:05:45.108671+0000","last_active":"2023-12-01T17:36:21.693835+0000","last_peered":"2023-12-01T17:36:21.693835+0000","last_clean":"2023-12-01T17:36:21.693835+0000","last_became_active":"2023-12-01T17:05:45.102496+0000","last_became_peered":"2023-12-01T17:05:45.102496+0000","last_unstale":"2023-12-01T17:36:21.693835+0000","last_undegraded":"2023-12-01T17:36:21.693835+0000","last_fullsized":"2023-12-01T17:36:21.693835+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:35:01.639680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.590006+0000","last_change":"2023-12-01T17:05:21.134670+0000","last_active":"2023-12-01T17:36:22.590006+0000","last_peered":"2023-12-01T17:36:22.590006+0000","last_clean":"2023-12-01T17:36:22.590006+0000","last_became_active":"2023-12-01T17:05:21.112377+0000","last_became_peered":"2023-12-01T17:05:21.112377+0000","last_unstale":"2023-12-01T17:36:22.590006+0000","last_undegraded":"2023-12-01T17:36:22.590006+0000","last_fullsized":"2023-12-01T17:36:22.590006+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:40:39.378498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.131517+0000","last_change":"2023-12-01T17:05:12.338713+0000","last_active":"2023-12-01T17:36:26.131517+0000","last_peered":"2023-12-01T17:36:26.131517+0000","last_clean":"2023-12-01T17:36:26.131517+0000","last_became_active":"2023-12-01T17:05:12.325443+0000","last_became_peered":"2023-12-01T17:05:12.325443+0000","last_unstale":"2023-12-01T17:36:26.131517+0000","last_undegraded":"2023-12-01T17:36:26.131517+0000","last_fullsized":"2023-12-01T17:36:26.131517+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:16:52.693300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.577087+0000","last_change":"2023-12-01T17:05:35.977136+0000","last_active":"2023-12-01T17:36:22.577087+0000","last_peered":"2023-12-01T17:36:22.577087+0000","last_clean":"2023-12-01T17:36:22.577087+0000","last_became_active":"2023-12-01T17:05:35.971286+0000","last_became_peered":"2023-12-01T17:05:35.971286+0000","last_unstale":"2023-12-01T17:36:22.577087+0000","last_undegraded":"2023-12-01T17:36:22.577087+0000","last_fullsized":"2023-12-01T17:36:22.577087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:40:27.793735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1046,"reported_epoch":1096,"state":"active+clean","last_fresh":"2023-12-01T17:36:25.149276+0000","last_change":"2023-12-01T17:05:33.458007+0000","last_active":"2023-12-01T17:36:25.149276+0000","last_peered":"2023-12-01T17:36:25.149276+0000","last_clean":"2023-12-01T17:36:25.149276+0000","last_became_active":"2023-12-01T17:05:33.448448+0000","last_became_peered":"2023-12-01T17:05:33.448448+0000","last_unstale":"2023-12-01T17:36:25.149276+0000","last_undegraded":"2023-12-01T17:36:25.149276+0000","last_fullsized":"2023-12-01T17:36:25.149276+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:03:13.718758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1051,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.606030+0000","last_change":"2023-12-01T17:05:36.335708+0000","last_active":"2023-12-01T17:36:22.606030+0000","last_peered":"2023-12-01T17:36:22.606030+0000","last_clean":"2023-12-01T17:36:22.606030+0000","last_became_active":"2023-12-01T17:05:36.329117+0000","last_became_peered":"2023-12-01T17:05:36.329117+0000","last_unstale":"2023-12-01T17:36:22.606030+0000","last_undegraded":"2023-12-01T17:36:22.606030+0000","last_fullsized":"2023-12-01T17:36:22.606030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T00:46:41.192583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.140840+0000","last_change":"2023-12-01T17:05:11.457400+0000","last_active":"2023-12-01T17:36:26.140840+0000","last_peered":"2023-12-01T17:36:26.140840+0000","last_clean":"2023-12-01T17:36:26.140840+0000","last_became_active":"2023-12-01T17:05:11.452081+0000","last_became_peered":"2023-12-01T17:05:11.452081+0000","last_unstale":"2023-12-01T17:36:26.140840+0000","last_undegraded":"2023-12-01T17:36:26.140840+0000","last_fullsized":"2023-12-01T17:36:26.140840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:10:31.762918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1051,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.526103+0000","last_change":"2023-12-01T17:05:36.195013+0000","last_active":"2023-12-01T17:36:21.526103+0000","last_peered":"2023-12-01T17:36:21.526103+0000","last_clean":"2023-12-01T17:36:21.526103+0000","last_became_active":"2023-12-01T17:05:36.185378+0000","last_became_peered":"2023-12-01T17:05:36.185378+0000","last_unstale":"2023-12-01T17:36:21.526103+0000","last_undegraded":"2023-12-01T17:36:21.526103+0000","last_fullsized":"2023-12-01T17:36:21.526103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:38:11.192241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":1159,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.489678+0000","last_change":"2023-12-01T17:03:54.894752+0000","last_active":"2023-12-01T17:36:22.489678+0000","last_peered":"2023-12-01T17:36:22.489678+0000","last_clean":"2023-12-01T17:36:22.489678+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:36:22.489678+0000","last_undegraded":"2023-12-01T17:36:22.489678+0000","last_fullsized":"2023-12-01T17:36:22.489678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_clean_scrub_stamp":"2023-12-01T17:03:51.457677+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:12:02.023301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1049,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.660940+0000","last_change":"2023-12-01T17:04:01.495183+0000","last_active":"2023-12-01T17:36:22.660940+0000","last_peered":"2023-12-01T17:36:22.660940+0000","last_clean":"2023-12-01T17:36:22.660940+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:36:22.660940+0000","last_undegraded":"2023-12-01T17:36:22.660940+0000","last_fullsized":"2023-12-01T17:36:22.660940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:13:19.704857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1044,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.563641+0000","last_change":"2023-12-01T17:05:15.674793+0000","last_active":"2023-12-01T17:36:22.563641+0000","last_peered":"2023-12-01T17:36:22.563641+0000","last_clean":"2023-12-01T17:36:22.563641+0000","last_became_active":"2023-12-01T17:05:15.659631+0000","last_became_peered":"2023-12-01T17:05:15.659631+0000","last_unstale":"2023-12-01T17:36:22.563641+0000","last_undegraded":"2023-12-01T17:36:22.563641+0000","last_fullsized":"2023-12-01T17:36:22.563641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:04:05.554372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1051,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.638318+0000","last_change":"2023-12-01T17:04:01.146575+0000","last_active":"2023-12-01T17:36:22.638318+0000","last_peered":"2023-12-01T17:36:22.638318+0000","last_clean":"2023-12-01T17:36:22.638318+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:36:22.638318+0000","last_undegraded":"2023-12-01T17:36:22.638318+0000","last_fullsized":"2023-12-01T17:36:22.638318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:18:50.459254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1047,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.513553+0000","last_change":"2023-12-01T17:05:11.335421+0000","last_active":"2023-12-01T17:36:22.513553+0000","last_peered":"2023-12-01T17:36:22.513553+0000","last_clean":"2023-12-01T17:36:22.513553+0000","last_became_active":"2023-12-01T17:05:11.329382+0000","last_became_peered":"2023-12-01T17:05:11.329382+0000","last_unstale":"2023-12-01T17:36:22.513553+0000","last_undegraded":"2023-12-01T17:36:22.513553+0000","last_fullsized":"2023-12-01T17:36:22.513553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:01:09.667175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1057,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.528202+0000","last_change":"2023-12-01T17:04:01.107520+0000","last_active":"2023-12-01T17:36:21.528202+0000","last_peered":"2023-12-01T17:36:21.528202+0000","last_clean":"2023-12-01T17:36:21.528202+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:36:21.528202+0000","last_undegraded":"2023-12-01T17:36:21.528202+0000","last_fullsized":"2023-12-01T17:36:21.528202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:47:47.456810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1053,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.143519+0000","last_change":"2023-12-01T17:05:16.493438+0000","last_active":"2023-12-01T17:36:26.143519+0000","last_peered":"2023-12-01T17:36:26.143519+0000","last_clean":"2023-12-01T17:36:26.143519+0000","last_became_active":"2023-12-01T17:05:16.481672+0000","last_became_peered":"2023-12-01T17:05:16.481672+0000","last_unstale":"2023-12-01T17:36:26.143519+0000","last_undegraded":"2023-12-01T17:36:26.143519+0000","last_fullsized":"2023-12-01T17:36:26.143519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:22:21.064006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1050,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.647330+0000","last_change":"2023-12-01T17:04:01.438358+0000","last_active":"2023-12-01T17:36:22.647330+0000","last_peered":"2023-12-01T17:36:22.647330+0000","last_clean":"2023-12-01T17:36:22.647330+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:36:22.647330+0000","last_undegraded":"2023-12-01T17:36:22.647330+0000","last_fullsized":"2023-12-01T17:36:22.647330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:45:24.963538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1050,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.533853+0000","last_change":"2023-12-01T17:05:14.953876+0000","last_active":"2023-12-01T17:36:21.533853+0000","last_peered":"2023-12-01T17:36:21.533853+0000","last_clean":"2023-12-01T17:36:21.533853+0000","last_became_active":"2023-12-01T17:05:14.948506+0000","last_became_peered":"2023-12-01T17:05:14.948506+0000","last_unstale":"2023-12-01T17:36:21.533853+0000","last_undegraded":"2023-12-01T17:36:21.533853+0000","last_fullsized":"2023-12-01T17:36:21.533853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:52.139356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1060,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.137101+0000","last_change":"2023-12-01T17:04:01.467571+0000","last_active":"2023-12-01T17:36:26.137101+0000","last_peered":"2023-12-01T17:36:26.137101+0000","last_clean":"2023-12-01T17:36:26.137101+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:36:26.137101+0000","last_undegraded":"2023-12-01T17:36:26.137101+0000","last_fullsized":"2023-12-01T17:36:26.137101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:41:15.693504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1047,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.486427+0000","last_change":"2023-12-01T17:05:11.984847+0000","last_active":"2023-12-01T17:36:22.486427+0000","last_peered":"2023-12-01T17:36:22.486427+0000","last_clean":"2023-12-01T17:36:22.486427+0000","last_became_active":"2023-12-01T17:05:11.693016+0000","last_became_peered":"2023-12-01T17:05:11.693016+0000","last_unstale":"2023-12-01T17:36:22.486427+0000","last_undegraded":"2023-12-01T17:36:22.486427+0000","last_fullsized":"2023-12-01T17:36:22.486427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:40:29.327298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1050,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.681781+0000","last_change":"2023-12-01T17:04:01.201853+0000","last_active":"2023-12-01T17:36:22.681781+0000","last_peered":"2023-12-01T17:36:22.681781+0000","last_clean":"2023-12-01T17:36:22.681781+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:36:22.681781+0000","last_undegraded":"2023-12-01T17:36:22.681781+0000","last_fullsized":"2023-12-01T17:36:22.681781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:38:19.219620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1063,"reported_epoch":1095,"state":"active+clean","last_fresh":"2023-12-01T17:36:23.675866+0000","last_change":"2023-12-01T17:05:12.792097+0000","last_active":"2023-12-01T17:36:23.675866+0000","last_peered":"2023-12-01T17:36:23.675866+0000","last_clean":"2023-12-01T17:36:23.675866+0000","last_became_active":"2023-12-01T17:05:12.768446+0000","last_became_peered":"2023-12-01T17:05:12.768446+0000","last_unstale":"2023-12-01T17:36:23.675866+0000","last_undegraded":"2023-12-01T17:36:23.675866+0000","last_fullsized":"2023-12-01T17:36:23.675866+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:17:04.335014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1063,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.719744+0000","last_change":"2023-12-01T17:04:01.536883+0000","last_active":"2023-12-01T17:36:21.719744+0000","last_peered":"2023-12-01T17:36:21.719744+0000","last_clean":"2023-12-01T17:36:21.719744+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:36:21.719744+0000","last_undegraded":"2023-12-01T17:36:21.719744+0000","last_fullsized":"2023-12-01T17:36:21.719744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:42:47.593805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1053,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.522414+0000","last_change":"2023-12-01T17:05:24.990862+0000","last_active":"2023-12-01T17:36:21.522414+0000","last_peered":"2023-12-01T17:36:21.522414+0000","last_clean":"2023-12-01T17:36:21.522414+0000","last_became_active":"2023-12-01T17:05:24.924547+0000","last_became_peered":"2023-12-01T17:05:24.924547+0000","last_unstale":"2023-12-01T17:36:21.522414+0000","last_undegraded":"2023-12-01T17:36:21.522414+0000","last_fullsized":"2023-12-01T17:36:21.522414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:58.899609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.698718+0000","last_change":"2023-12-01T17:04:01.476480+0000","last_active":"2023-12-01T17:36:22.698718+0000","last_peered":"2023-12-01T17:36:22.698718+0000","last_clean":"2023-12-01T17:36:22.698718+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:36:22.698718+0000","last_undegraded":"2023-12-01T17:36:22.698718+0000","last_fullsized":"2023-12-01T17:36:22.698718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:26:16.264231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1059,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.695845+0000","last_change":"2023-12-01T17:05:28.757137+0000","last_active":"2023-12-01T17:36:21.695845+0000","last_peered":"2023-12-01T17:36:21.695845+0000","last_clean":"2023-12-01T17:36:21.695845+0000","last_became_active":"2023-12-01T17:05:28.748541+0000","last_became_peered":"2023-12-01T17:05:28.748541+0000","last_unstale":"2023-12-01T17:36:21.695845+0000","last_undegraded":"2023-12-01T17:36:21.695845+0000","last_fullsized":"2023-12-01T17:36:21.695845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:15:33.293632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1","version":"62'7","reported_seq":1064,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.578985+0000","last_change":"2023-12-01T17:05:38.783874+0000","last_active":"2023-12-01T17:36:22.578985+0000","last_peered":"2023-12-01T17:36:22.578985+0000","last_clean":"2023-12-01T17:36:22.578985+0000","last_became_active":"2023-12-01T17:05:38.775069+0000","last_became_peered":"2023-12-01T17:05:38.775069+0000","last_unstale":"2023-12-01T17:36:22.578985+0000","last_undegraded":"2023-12-01T17:36:22.578985+0000","last_fullsized":"2023-12-01T17:36:22.578985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:09:31.642602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1052,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.565789+0000","last_change":"2023-12-01T17:05:17.531869+0000","last_active":"2023-12-01T17:36:22.565789+0000","last_peered":"2023-12-01T17:36:22.565789+0000","last_clean":"2023-12-01T17:36:22.565789+0000","last_became_active":"2023-12-01T17:05:17.447969+0000","last_became_peered":"2023-12-01T17:05:17.447969+0000","last_unstale":"2023-12-01T17:36:22.565789+0000","last_undegraded":"2023-12-01T17:36:22.565789+0000","last_fullsized":"2023-12-01T17:36:22.565789+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:20:05.487052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.496765+0000","last_change":"2023-12-01T17:05:12.380756+0000","last_active":"2023-12-01T17:36:22.496765+0000","last_peered":"2023-12-01T17:36:22.496765+0000","last_clean":"2023-12-01T17:36:22.496765+0000","last_became_active":"2023-12-01T17:05:12.334114+0000","last_became_peered":"2023-12-01T17:05:12.334114+0000","last_unstale":"2023-12-01T17:36:22.496765+0000","last_undegraded":"2023-12-01T17:36:22.496765+0000","last_fullsized":"2023-12-01T17:36:22.496765+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T00:52:19.170433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1065,"reported_epoch":1095,"state":"active+clean","last_fresh":"2023-12-01T17:36:23.690252+0000","last_change":"2023-12-01T17:05:39.346000+0000","last_active":"2023-12-01T17:36:23.690252+0000","last_peered":"2023-12-01T17:36:23.690252+0000","last_clean":"2023-12-01T17:36:23.690252+0000","last_became_active":"2023-12-01T17:05:39.340484+0000","last_became_peered":"2023-12-01T17:05:39.340484+0000","last_unstale":"2023-12-01T17:36:23.690252+0000","last_undegraded":"2023-12-01T17:36:23.690252+0000","last_fullsized":"2023-12-01T17:36:23.690252+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:50:46.835583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.517739+0000","last_change":"2023-12-01T17:05:58.777840+0000","last_active":"2023-12-01T17:36:22.517739+0000","last_peered":"2023-12-01T17:36:22.517739+0000","last_clean":"2023-12-01T17:36:22.517739+0000","last_became_active":"2023-12-01T17:05:58.772230+0000","last_became_peered":"2023-12-01T17:05:58.772230+0000","last_unstale":"2023-12-01T17:36:22.517739+0000","last_undegraded":"2023-12-01T17:36:22.517739+0000","last_fullsized":"2023-12-01T17:36:22.517739+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:05:58.781340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1044,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.667139+0000","last_change":"2023-12-01T17:05:11.225296+0000","last_active":"2023-12-01T17:36:22.667139+0000","last_peered":"2023-12-01T17:36:22.667139+0000","last_clean":"2023-12-01T17:36:22.667139+0000","last_became_active":"2023-12-01T17:05:11.209283+0000","last_became_peered":"2023-12-01T17:05:11.209283+0000","last_unstale":"2023-12-01T17:36:22.667139+0000","last_undegraded":"2023-12-01T17:36:22.667139+0000","last_fullsized":"2023-12-01T17:36:22.667139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:39:18.388303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.566053+0000","last_change":"2023-12-01T17:05:17.271434+0000","last_active":"2023-12-01T17:36:22.566053+0000","last_peered":"2023-12-01T17:36:22.566053+0000","last_clean":"2023-12-01T17:36:22.566053+0000","last_became_active":"2023-12-01T17:05:17.215461+0000","last_became_peered":"2023-12-01T17:05:17.215461+0000","last_unstale":"2023-12-01T17:36:22.566053+0000","last_undegraded":"2023-12-01T17:36:22.566053+0000","last_fullsized":"2023-12-01T17:36:22.566053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:14:02.813273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1044,"reported_epoch":1096,"state":"active+clean","last_fresh":"2023-12-01T17:36:25.151563+0000","last_change":"2023-12-01T17:05:21.697504+0000","last_active":"2023-12-01T17:36:25.151563+0000","last_peered":"2023-12-01T17:36:25.151563+0000","last_clean":"2023-12-01T17:36:25.151563+0000","last_became_active":"2023-12-01T17:05:21.668214+0000","last_became_peered":"2023-12-01T17:05:21.668214+0000","last_unstale":"2023-12-01T17:36:25.151563+0000","last_undegraded":"2023-12-01T17:36:25.151563+0000","last_fullsized":"2023-12-01T17:36:25.151563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:07:59.125552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1060,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.688026+0000","last_change":"2023-12-01T17:05:46.022392+0000","last_active":"2023-12-01T17:36:21.688026+0000","last_peered":"2023-12-01T17:36:21.688026+0000","last_clean":"2023-12-01T17:36:21.688026+0000","last_became_active":"2023-12-01T17:05:46.016610+0000","last_became_peered":"2023-12-01T17:05:46.016610+0000","last_unstale":"2023-12-01T17:36:21.688026+0000","last_undegraded":"2023-12-01T17:36:21.688026+0000","last_fullsized":"2023-12-01T17:36:21.688026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:47:36.167822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.636608+0000","last_change":"2023-12-01T17:05:37.365267+0000","last_active":"2023-12-01T17:36:22.636608+0000","last_peered":"2023-12-01T17:36:22.636608+0000","last_clean":"2023-12-01T17:36:22.636608+0000","last_became_active":"2023-12-01T17:05:37.358444+0000","last_became_peered":"2023-12-01T17:05:37.358444+0000","last_unstale":"2023-12-01T17:36:22.636608+0000","last_undegraded":"2023-12-01T17:36:22.636608+0000","last_fullsized":"2023-12-01T17:36:22.636608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:48:13.502545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1055,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.162048+0000","last_change":"2023-12-01T17:05:13.509587+0000","last_active":"2023-12-01T17:36:26.162048+0000","last_peered":"2023-12-01T17:36:26.162048+0000","last_clean":"2023-12-01T17:36:26.162048+0000","last_became_active":"2023-12-01T17:05:13.499129+0000","last_became_peered":"2023-12-01T17:05:13.499129+0000","last_unstale":"2023-12-01T17:36:26.162048+0000","last_undegraded":"2023-12-01T17:36:26.162048+0000","last_fullsized":"2023-12-01T17:36:26.162048+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:25:34.541978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.649060+0000","last_change":"2023-12-01T17:05:37.748101+0000","last_active":"2023-12-01T17:36:22.649060+0000","last_peered":"2023-12-01T17:36:22.649060+0000","last_clean":"2023-12-01T17:36:22.649060+0000","last_became_active":"2023-12-01T17:05:37.740514+0000","last_became_peered":"2023-12-01T17:05:37.740514+0000","last_unstale":"2023-12-01T17:36:22.649060+0000","last_undegraded":"2023-12-01T17:36:22.649060+0000","last_fullsized":"2023-12-01T17:36:22.649060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T04:17:18.757175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444421,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71768,"kb_used_data":8028,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300072,"statfs":{"total":96636764160,"available":96563273728,"internally_reserved":0,"allocated":8220672,"data_stored":4735665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":20.553999999999998,"5min":12.125999999999999,"15min":15.801},"min":{"1min":6.5620000000000003,"5min":4.5899999999999999,"15min":4.5899999999999999},"max":{"1min":127.741,"5min":127.741,"15min":150.279},"last":8.2080000000000002},{"interface":"front","average":{"1min":29.547000000000001,"5min":14.099,"15min":17.922999999999998},"min":{"1min":8.5419999999999998,"5min":4.7229999999999999,"15min":4.7229999999999999},"max":{"1min":184.30600000000001,"5min":184.30600000000001,"15min":184.30600000000001},"last":10.763}]},{"osd":2,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":18.321999999999999,"5min":12.057,"15min":16.654},"min":{"1min":6.5709999999999997,"5min":5.4610000000000003,"15min":5.4610000000000003},"max":{"1min":99.156000000000006,"5min":99.156000000000006,"15min":208.18199999999999},"last":8.8569999999999993},{"interface":"front","average":{"1min":34.377000000000002,"5min":16.986999999999998,"15min":19.702000000000002},"min":{"1min":7.9260000000000002,"5min":6.8769999999999998,"15min":6.2030000000000003},"max":{"1min":216.785,"5min":216.785,"15min":216.785},"last":12.122999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":28.09,"5min":16.707999999999998,"15min":21.498000000000001},"min":{"1min":8.9960000000000004,"5min":7.8550000000000004,"15min":7.8550000000000004},"max":{"1min":243.88999999999999,"5min":243.88999999999999,"15min":276.07400000000001},"last":16.044},{"interface":"front","average":{"1min":30.187000000000001,"5min":19.353999999999999,"15min":24.324000000000002},"min":{"1min":13.336,"5min":10.973000000000001,"15min":7.2649999999999997},"max":{"1min":239.40000000000001,"5min":239.40000000000001,"15min":252.43299999999999},"last":36.654000000000003}]},{"osd":4,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":31.789000000000001,"5min":16.448,"15min":17.77},"min":{"1min":8.0429999999999993,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":230.511,"5min":230.511,"15min":230.511},"last":11.507999999999999},{"interface":"front","average":{"1min":33.975000000000001,"5min":17.349,"15min":21.777999999999999},"min":{"1min":7.6379999999999999,"5min":7.3979999999999997,"15min":7.3979999999999997},"max":{"1min":241.28100000000001,"5min":241.28100000000001,"15min":241.28100000000001},"last":9.6859999999999999}]},{"osd":5,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":27.834,"5min":16.713999999999999,"15min":18.786000000000001},"min":{"1min":6.859,"5min":6.859,"15min":6.149},"max":{"1min":191.78700000000001,"5min":191.78700000000001,"15min":191.78700000000001},"last":16.759},{"interface":"front","average":{"1min":17.850999999999999,"5min":15.477,"15min":22.440999999999999},"min":{"1min":7.3769999999999998,"5min":7.3769999999999998,"15min":7.0270000000000001},"max":{"1min":111.208,"5min":111.208,"15min":311.13299999999998},"last":13.367000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":22.614999999999998,"5min":17.094999999999999,"15min":18.754999999999999},"min":{"1min":11.262,"5min":10.199,"15min":7.0960000000000001},"max":{"1min":94.787999999999997,"5min":94.787999999999997,"15min":99.623000000000005},"last":33.271000000000001},{"interface":"front","average":{"1min":26.064,"5min":17.611000000000001,"15min":21.722999999999999},"min":{"1min":8.6600000000000001,"5min":8.6080000000000005,"15min":7.9909999999999997},"max":{"1min":153.83799999999999,"5min":153.83799999999999,"15min":186.11000000000001},"last":31.146000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":35.716000000000001,"5min":20.347000000000001,"15min":24.768999999999998},"min":{"1min":12.414999999999999,"5min":9.4819999999999993,"15min":9.1150000000000002},"max":{"1min":207.64599999999999,"5min":207.64599999999999,"15min":207.64599999999999},"last":35.598999999999997},{"interface":"front","average":{"1min":23.535,"5min":17.969999999999999,"15min":21.334},"min":{"1min":11.978999999999999,"5min":9.4469999999999992,"15min":9.0069999999999997},"max":{"1min":74.777000000000001,"5min":74.777000000000001,"15min":148.89599999999999},"last":14.211}]}]},{"osd":3,"up_from":16,"seq":68719477125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70992,"kb_used_data":7228,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300848,"statfs":{"total":96636764160,"available":96564068352,"internally_reserved":0,"allocated":7401472,"data_stored":4081800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":38.427,"5min":15.26,"15min":18.015000000000001},"min":{"1min":6.8440000000000003,"5min":5.5540000000000003,"15min":5.2539999999999996},"max":{"1min":180.16,"5min":180.16,"15min":181.94200000000001},"last":10.919},{"interface":"front","average":{"1min":28.507000000000001,"5min":13.943,"15min":18.838000000000001},"min":{"1min":4.915,"5min":4.915,"15min":4.915},"max":{"1min":107.5,"5min":107.5,"15min":427.36500000000001},"last":13.970000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":30.577999999999999,"5min":15.76,"15min":20.832999999999998},"min":{"1min":8.1419999999999995,"5min":7.585,"15min":6.4039999999999999},"max":{"1min":146.48099999999999,"5min":146.48099999999999,"15min":227.309},"last":7.3890000000000002},{"interface":"front","average":{"1min":33.158000000000001,"5min":15.33,"15min":19.213999999999999},"min":{"1min":8.2870000000000008,"5min":6.9850000000000003,"15min":5.827},"max":{"1min":223.179,"5min":223.179,"15min":421.40600000000001},"last":9.0570000000000004}]},{"osd":2,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":20.815999999999999,"5min":13.278,"15min":22.062000000000001},"min":{"1min":7.9829999999999997,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":86.969999999999999,"5min":86.969999999999999,"15min":451.97899999999998},"last":16.805},{"interface":"front","average":{"1min":29.763999999999999,"5min":15.894,"15min":23.613},"min":{"1min":6.8540000000000001,"5min":6.7190000000000003,"15min":6.5170000000000003},"max":{"1min":223.84,"5min":223.84,"15min":384.58699999999999},"last":14.712999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":33.475999999999999,"5min":17.899999999999999,"15min":21.863},"min":{"1min":9.6440000000000001,"5min":8.5640000000000001,"15min":6.5330000000000004},"max":{"1min":244.989,"5min":244.989,"15min":424.09500000000003},"last":20.486000000000001},{"interface":"front","average":{"1min":31.292000000000002,"5min":17.065000000000001,"15min":21.056000000000001},"min":{"1min":8.9700000000000006,"5min":7.9870000000000001,"15min":7.0049999999999999},"max":{"1min":159.78200000000001,"5min":159.78200000000001,"15min":202.70500000000001},"last":11.794}]},{"osd":5,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":32.497,"5min":19.154,"15min":25.114000000000001},"min":{"1min":10.481999999999999,"5min":9.7400000000000002,"15min":9.0860000000000003},"max":{"1min":228.96700000000001,"5min":228.96700000000001,"15min":424.80099999999999},"last":19.100999999999999},{"interface":"front","average":{"1min":23.297999999999998,"5min":15.457000000000001,"15min":21.076000000000001},"min":{"1min":8.0359999999999996,"5min":7.2850000000000001,"15min":7.1710000000000003},"max":{"1min":87.635000000000005,"5min":87.635000000000005,"15min":231.65199999999999},"last":18.146000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":27.949000000000002,"5min":17.577000000000002,"15min":21.687999999999999},"min":{"1min":8.2859999999999996,"5min":8.2859999999999996,"15min":7.9340000000000002},"max":{"1min":136.684,"5min":136.684,"15min":203.99100000000001},"last":10.311999999999999},{"interface":"front","average":{"1min":30.741,"5min":19.934000000000001,"15min":25.875},"min":{"1min":12.468999999999999,"5min":10.978,"15min":10.978},"max":{"1min":147.88200000000001,"5min":147.88200000000001,"15min":203.34100000000001},"last":13.345000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":25.036000000000001,"5min":16.992999999999999,"15min":22.780000000000001},"min":{"1min":8.7509999999999994,"5min":7.9589999999999996,"15min":7.0259999999999998},"max":{"1min":105.342,"5min":105.342,"15min":196.05699999999999},"last":22.114999999999998},{"interface":"front","average":{"1min":27.623000000000001,"5min":19.024999999999999,"15min":25.555},"min":{"1min":11.186,"5min":11.186,"15min":10.492000000000001},"max":{"1min":102.29000000000001,"5min":102.29000000000001,"15min":236.84700000000001},"last":23.364000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477123,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70996,"kb_used_data":7232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300844,"statfs":{"total":96636764160,"available":96564064256,"internally_reserved":0,"allocated":7405568,"data_stored":4081805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.219000000000001,"5min":14.260999999999999,"15min":20.582999999999998},"min":{"1min":6.4989999999999997,"5min":4.7110000000000003,"15min":4.6059999999999999},"max":{"1min":169.971,"5min":169.971,"15min":231.76499999999999},"last":9.3409999999999993},{"interface":"front","average":{"1min":29.649999999999999,"5min":14.412000000000001,"15min":19.628},"min":{"1min":5.3049999999999997,"5min":5.1890000000000001,"15min":4.8810000000000002},"max":{"1min":150.74199999999999,"5min":150.74199999999999,"15min":222.62},"last":10.647}]},{"osd":1,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":23.131,"5min":12.741,"15min":20.030000000000001},"min":{"1min":6.4980000000000002,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":182.83600000000001,"5min":182.83600000000001,"15min":220.96700000000001},"last":8.6430000000000007},{"interface":"front","average":{"1min":27.039999999999999,"5min":16.943999999999999,"15min":23.343},"min":{"1min":8.6869999999999994,"5min":8.6869999999999994,"15min":7.8639999999999999},"max":{"1min":196.886,"5min":196.886,"15min":213.27799999999999},"last":12.291}]},{"osd":3,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":40.603999999999999,"5min":17.788,"15min":22.448},"min":{"1min":7.0609999999999999,"5min":7.0609999999999999,"15min":6.734},"max":{"1min":177.04900000000001,"5min":177.04900000000001,"15min":219.51300000000001},"last":13.166},{"interface":"front","average":{"1min":41.942999999999998,"5min":19.474,"15min":24.347000000000001},"min":{"1min":7.6740000000000004,"5min":7.6740000000000004,"15min":6.9900000000000002},"max":{"1min":204.721,"5min":204.721,"15min":204.721},"last":19.106999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.542000000000002,"5min":16.068000000000001,"15min":21.274000000000001},"min":{"1min":8.6310000000000002,"5min":7.2060000000000004,"15min":6.6799999999999997},"max":{"1min":180.369,"5min":180.369,"15min":251.745},"last":26.879000000000001},{"interface":"front","average":{"1min":33.243000000000002,"5min":18.972000000000001,"15min":24.882999999999999},"min":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":7.5720000000000001},"max":{"1min":170.637,"5min":170.637,"15min":287.32100000000003},"last":25.776}]},{"osd":5,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":37.953000000000003,"5min":18.436,"15min":24.466999999999999},"min":{"1min":8.5060000000000002,"5min":8.3300000000000001,"15min":8.0020000000000007},"max":{"1min":206.56700000000001,"5min":206.56700000000001,"15min":210.012},"last":11.662000000000001},{"interface":"front","average":{"1min":33.348999999999997,"5min":19.402000000000001,"15min":25.024000000000001},"min":{"1min":9.1980000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"max":{"1min":232.91900000000001,"5min":232.91900000000001,"15min":232.91900000000001},"last":14.550000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":35.573999999999998,"5min":19.463000000000001,"15min":23.882999999999999},"min":{"1min":8.8710000000000004,"5min":8.8710000000000004,"15min":8.1829999999999998},"max":{"1min":182.07400000000001,"5min":182.07400000000001,"15min":237.21899999999999},"last":17.206},{"interface":"front","average":{"1min":38.033999999999999,"5min":20.099,"15min":27.759},"min":{"1min":12.598000000000001,"5min":9.5960000000000001,"15min":9.5649999999999995},"max":{"1min":185.90299999999999,"5min":185.90299999999999,"15min":241.92699999999999},"last":17.835000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":19.201000000000001,"5min":16.905000000000001,"15min":24.222000000000001},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.5690000000000008},"max":{"1min":68.075000000000003,"5min":68.075000000000003,"15min":341.73000000000002},"last":15.349},{"interface":"front","average":{"1min":22.43,"5min":18.937999999999999,"15min":27.407},"min":{"1min":10.853,"5min":10.853,"15min":10.853},"max":{"1min":89.631,"5min":89.631,"15min":343.17399999999998},"last":16.285}]}]},{"osd":0,"up_from":15,"seq":64424509828,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70588,"kb_used_data":7252,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301252,"statfs":{"total":96636764160,"available":96564482048,"internally_reserved":0,"allocated":7426048,"data_stored":4096103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 17:36:15 2023","interfaces":[{"interface":"back","average":{"1min":11.167,"5min":10.766999999999999,"15min":17.079000000000001},"min":{"1min":5.5579999999999998,"5min":5.5579999999999998,"15min":4.3550000000000004},"max":{"1min":33.322000000000003,"5min":33.322000000000003,"15min":210.43299999999999},"last":66.680999999999997},{"interface":"front","average":{"1min":12.661,"5min":10.788,"15min":18.704999999999998},"min":{"1min":6.242,"5min":6.0750000000000002,"15min":5.2590000000000003},"max":{"1min":44.020000000000003,"5min":44.020000000000003,"15min":257.30500000000001},"last":92.177000000000007}]},{"osd":2,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.808,"5min":14.16,"15min":19.648},"min":{"1min":8.2010000000000005,"5min":7.4740000000000002,"15min":6.452},"max":{"1min":94.155000000000001,"5min":94.155000000000001,"15min":239.24600000000001},"last":94.155000000000001},{"interface":"front","average":{"1min":16.765000000000001,"5min":13.634,"15min":18.114999999999998},"min":{"1min":8.3179999999999996,"5min":6.8579999999999997,"15min":5.6520000000000001},"max":{"1min":86.099999999999994,"5min":86.099999999999994,"15min":246.28299999999999},"last":86.099999999999994}]},{"osd":3,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":12.295999999999999,"15min":19.577000000000002},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"max":{"1min":128.08099999999999,"5min":128.08099999999999,"15min":240.017},"last":53.909999999999997},{"interface":"front","average":{"1min":28.396000000000001,"5min":17.071999999999999,"15min":25.015999999999998},"min":{"1min":9.452,"5min":8.3499999999999996,"15min":7.2779999999999996},"max":{"1min":136.40600000000001,"5min":136.40600000000001,"15min":249.86099999999999},"last":93.147000000000006}]},{"osd":4,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":15.007999999999999,"5min":12.295,"15min":18.792999999999999},"min":{"1min":7.5780000000000003,"5min":6.6909999999999998,"15min":6.6909999999999998},"max":{"1min":77.429000000000002,"5min":77.429000000000002,"15min":241.292},"last":196.71799999999999},{"interface":"front","average":{"1min":19.893999999999998,"5min":16.827000000000002,"15min":22.696000000000002},"min":{"1min":11.875,"5min":9.3759999999999994,"15min":7.2290000000000001},"max":{"1min":91.792000000000002,"5min":91.792000000000002,"15min":236.65799999999999},"last":207.38}]},{"osd":5,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":16.934000000000001,"5min":14.622,"15min":24.321999999999999},"min":{"1min":9.2949999999999999,"5min":8.6929999999999996,"15min":7.9400000000000004},"max":{"1min":45.228000000000002,"5min":45.228000000000002,"15min":241.91200000000001},"last":55.027000000000001},{"interface":"front","average":{"1min":16.408000000000001,"5min":14.084,"15min":21.462},"min":{"1min":7.2779999999999996,"5min":7.2779999999999996,"15min":6.4279999999999999},"max":{"1min":66.477000000000004,"5min":66.477000000000004,"15min":247.185},"last":73.453000000000003}]},{"osd":6,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":15.752000000000001,"15min":23.873999999999999},"min":{"1min":9.8810000000000002,"5min":8.2829999999999995,"15min":7.923},"max":{"1min":50.194000000000003,"5min":50.194000000000003,"15min":244.93799999999999},"last":68.665999999999997},{"interface":"front","average":{"1min":16.581,"5min":15.106999999999999,"15min":22.677},"min":{"1min":9.2609999999999992,"5min":9.2110000000000003,"15min":7.734},"max":{"1min":42.387999999999998,"5min":42.387999999999998,"15min":247.80500000000001},"last":83.817999999999998}]},{"osd":7,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":18.469999999999999,"5min":17.091999999999999,"15min":24.052},"min":{"1min":11.074999999999999,"5min":10.281000000000001,"15min":9.0820000000000007},"max":{"1min":37.758000000000003,"5min":37.758000000000003,"15min":238.58099999999999},"last":199.01400000000001},{"interface":"front","average":{"1min":18.225000000000001,"5min":16.898,"15min":24.481000000000002},"min":{"1min":11.706,"5min":9.3140000000000001,"15min":9.3140000000000001},"max":{"1min":36.850000000000001,"5min":36.850000000000001,"15min":273.142},"last":91.361000000000004}]}]},{"osd":6,"up_from":14,"seq":60129542534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70568,"kb_used_data":7232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301272,"statfs":{"total":96636764160,"available":96564502528,"internally_reserved":0,"allocated":7405568,"data_stored":4081805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":10.34,"5min":9.7390000000000008,"15min":13.387},"min":{"1min":4.8339999999999996,"5min":4.577,"15min":4.0730000000000004},"max":{"1min":25.731999999999999,"5min":25.731999999999999,"15min":129.89699999999999},"last":6.7709999999999999},{"interface":"front","average":{"1min":11.954000000000001,"5min":10.832000000000001,"15min":15.518000000000001},"min":{"1min":6.5010000000000003,"5min":6.4669999999999996,"15min":4.8129999999999997},"max":{"1min":34.526000000000003,"5min":34.526000000000003,"15min":155.99100000000001},"last":12.582000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":14.457000000000001,"5min":12.138,"15min":15.375},"min":{"1min":7.9969999999999999,"5min":6.702,"15min":6.2789999999999999},"max":{"1min":50.130000000000003,"5min":50.130000000000003,"15min":175.55000000000001},"last":10.417},{"interface":"front","average":{"1min":18.449000000000002,"5min":15.638999999999999,"15min":19.158000000000001},"min":{"1min":10.378,"5min":10.378,"15min":7.0490000000000004},"max":{"1min":54.060000000000002,"5min":54.060000000000002,"15min":130.91999999999999},"last":11.733000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":21.776,"5min":15.327,"15min":19.893999999999998},"min":{"1min":7.0010000000000003,"5min":6.8879999999999999,"15min":6.8879999999999999},"max":{"1min":172.459,"5min":172.459,"15min":191.71299999999999},"last":13.83},{"interface":"front","average":{"1min":19.948,"5min":11.276999999999999,"15min":16.713000000000001},"min":{"1min":4.4370000000000003,"5min":4.4370000000000003,"15min":4.4370000000000003},"max":{"1min":174.608,"5min":174.608,"15min":244.46299999999999},"last":5.1449999999999996}]},{"osd":3,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":19.504999999999999,"5min":15.193,"15min":20.762},"min":{"1min":10.653,"5min":8.2929999999999993,"15min":6.5830000000000002},"max":{"1min":70.066999999999993,"5min":70.066999999999993,"15min":223.97800000000001},"last":16.021999999999998},{"interface":"front","average":{"1min":18.291,"5min":14.847,"15min":22.268000000000001},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":6.8559999999999999},"max":{"1min":71.659999999999997,"5min":71.659999999999997,"15min":212.309},"last":17.539999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.582000000000001,"5min":14.68,"15min":19.289999999999999},"min":{"1min":10.567,"5min":8.952,"15min":7.0510000000000002},"max":{"1min":21.972000000000001,"5min":22.463000000000001,"15min":134.95699999999999},"last":15.204000000000001},{"interface":"front","average":{"1min":17.640999999999998,"5min":17.771000000000001,"15min":22.725000000000001},"min":{"1min":11.27,"5min":11.27,"15min":9.9809999999999999},"max":{"1min":22.663,"5min":22.663,"15min":135.798},"last":19.507000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.782,"5min":15.356,"15min":20.988},"min":{"1min":7.8040000000000003,"5min":7.8040000000000003,"15min":7.2869999999999999},"max":{"1min":20.332000000000001,"5min":20.966000000000001,"15min":288.577},"last":13.218999999999999},{"interface":"front","average":{"1min":17.925999999999998,"5min":17.777000000000001,"15min":23.446999999999999},"min":{"1min":12.861000000000001,"5min":12.124000000000001,"15min":10.234},"max":{"1min":23.948,"5min":23.948,"15min":142.77699999999999},"last":18.559999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":11.468999999999999,"5min":10.596,"15min":15.210000000000001},"min":{"1min":7.2649999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"max":{"1min":21.209,"5min":21.209,"15min":168.59700000000001},"last":9.2189999999999994},{"interface":"front","average":{"1min":17.047000000000001,"5min":16.718,"15min":21.393000000000001},"min":{"1min":11.954000000000001,"5min":8.9459999999999997,"15min":8.4770000000000003},"max":{"1min":24.709,"5min":25.518000000000001,"15min":183.054},"last":16.818000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542535,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74332,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297508,"statfs":{"total":96636764160,"available":96560648192,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":24.530999999999999,"5min":11.712,"15min":15.721},"min":{"1min":4.3620000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"max":{"1min":201.333,"5min":201.333,"15min":228.261},"last":6.4809999999999999},{"interface":"front","average":{"1min":27.792000000000002,"5min":13.209,"15min":17.690999999999999},"min":{"1min":6.5709999999999997,"5min":5.0339999999999998,"15min":4.6580000000000004},"max":{"1min":221.876,"5min":221.876,"15min":222.441},"last":16.992000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.556000000000001,"5min":12.199999999999999,"15min":19.036000000000001},"min":{"1min":7.0490000000000004,"5min":6.0640000000000001,"15min":4.8899999999999997},"max":{"1min":66.040999999999997,"5min":66.040999999999997,"15min":216.20099999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":17.574000000000002,"5min":12.794,"15min":19.777000000000001},"min":{"1min":7.7430000000000003,"5min":6.5650000000000004,"15min":5.6210000000000004},"max":{"1min":65.156999999999996,"5min":65.156999999999996,"15min":255.95599999999999},"last":15.778}]},{"osd":2,"last update":"Fri Dec 1 17:35:58 2023","interfaces":[{"interface":"back","average":{"1min":12.641999999999999,"5min":11.792,"15min":17.571999999999999},"min":{"1min":6.468,"5min":6.468,"15min":6.468},"max":{"1min":39.947000000000003,"5min":39.947000000000003,"15min":179.81999999999999},"last":7.2530000000000001},{"interface":"front","average":{"1min":16.623999999999999,"5min":15.505000000000001,"15min":22.356999999999999},"min":{"1min":8.7070000000000007,"5min":8.7070000000000007,"15min":8.7070000000000007},"max":{"1min":42.759,"5min":42.759,"15min":224.47300000000001},"last":11.065}]},{"osd":3,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":13.112,"15min":19.452999999999999},"min":{"1min":6.819,"5min":6.819,"15min":5.9020000000000001},"max":{"1min":61.933,"5min":61.933,"15min":248.07300000000001},"last":19.225999999999999},{"interface":"front","average":{"1min":22.122,"5min":16.693000000000001,"15min":23.396000000000001},"min":{"1min":11.053000000000001,"5min":10.221,"15min":9.7989999999999995},"max":{"1min":62.886000000000003,"5min":62.886000000000003,"15min":235.02699999999999},"last":18.350999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":17.068999999999999,"5min":15.241,"15min":19.116},"min":{"1min":10.965999999999999,"5min":9.4930000000000003,"15min":8.1699999999999999},"max":{"1min":59.747,"5min":59.747,"15min":211.09399999999999},"last":9.8529999999999998},{"interface":"front","average":{"1min":14.052,"5min":14.444000000000001,"15min":18.030000000000001},"min":{"1min":9.6010000000000009,"5min":8.6850000000000005,"15min":7.0540000000000003},"max":{"1min":29.402000000000001,"5min":29.402000000000001,"15min":216.85300000000001},"last":12.993}]},{"osd":6,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":20.969999999999999,"5min":17.178000000000001,"15min":21.32},"min":{"1min":11.702,"5min":10.103,"15min":8.4339999999999993},"max":{"1min":111.396,"5min":111.396,"15min":223.89500000000001},"last":16.402999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":14.01,"15min":14.154999999999999},"min":{"1min":7.5949999999999998,"5min":5.6829999999999998,"15min":5.6479999999999997},"max":{"1min":34.613,"5min":34.613,"15min":159.84399999999999},"last":13.773}]},{"osd":7,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":18.885999999999999,"5min":17.652000000000001,"15min":21.867000000000001},"min":{"1min":10.467000000000001,"5min":10.467000000000001,"15min":9.6430000000000007},"max":{"1min":62.256999999999998,"5min":62.256999999999998,"15min":226.06100000000001},"last":14.355},{"interface":"front","average":{"1min":21.974,"5min":17.885000000000002,"15min":21.588999999999999},"min":{"1min":9.3369999999999997,"5min":9.3369999999999997,"15min":8.1820000000000004},"max":{"1min":125.431,"5min":125.431,"15min":317.267},"last":12.250999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542535,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75536,"kb_used_data":7636,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296304,"statfs":{"total":96636764160,"available":96559415296,"internally_reserved":0,"allocated":7819264,"data_stored":4337825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":7.8300000000000001,"5min":9.3949999999999996,"15min":16.763000000000002},"min":{"1min":4.1109999999999998,"5min":4.1109999999999998,"15min":3.7690000000000001},"max":{"1min":15.778,"5min":24.963999999999999,"15min":521.55899999999997},"last":4.609},{"interface":"front","average":{"1min":9.3529999999999998,"5min":9.8160000000000007,"15min":15.084},"min":{"1min":6.3360000000000003,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":13.864000000000001,"5min":21.491,"15min":281.125},"last":9.0220000000000002}]},{"osd":1,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":11.353999999999999,"5min":11.315,"15min":19.023},"min":{"1min":8.2140000000000004,"5min":6.7969999999999997,"15min":6.2770000000000001},"max":{"1min":14.935,"5min":21.539000000000001,"15min":308.99299999999999},"last":10.351000000000001},{"interface":"front","average":{"1min":16.753,"5min":15.666,"15min":22.716000000000001},"min":{"1min":10.832000000000001,"5min":9.1069999999999993,"15min":6.3079999999999998},"max":{"1min":31.140000000000001,"5min":31.140000000000001,"15min":289.79399999999998},"last":11.068}]},{"osd":2,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":16.849,"5min":15.061999999999999,"15min":21.503},"min":{"1min":11.24,"5min":9.1029999999999998,"15min":5.976},"max":{"1min":27.797999999999998,"5min":27.797999999999998,"15min":298.18200000000002},"last":14.058999999999999},{"interface":"front","average":{"1min":12.192,"5min":10.987,"15min":16.457999999999998},"min":{"1min":8.0640000000000001,"5min":6.4820000000000002,"15min":6.4820000000000002},"max":{"1min":25.771999999999998,"5min":25.771999999999998,"15min":223.768},"last":11.661}]},{"osd":3,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":12.680999999999999,"15min":18.734000000000002},"min":{"1min":8.6850000000000005,"5min":7.1580000000000004,"15min":6.2169999999999996},"max":{"1min":26.462,"5min":26.462,"15min":243.40299999999999},"last":15.862},{"interface":"front","average":{"1min":17.468,"5min":16.356999999999999,"15min":22.004000000000001},"min":{"1min":13.864000000000001,"5min":10.156000000000001,"15min":8.4390000000000001},"max":{"1min":37.262999999999998,"5min":37.262999999999998,"15min":253.56100000000001},"last":17.404}]},{"osd":5,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":17.428999999999998,"5min":15.695,"15min":22.879999999999999},"min":{"1min":10.804,"5min":9.5199999999999996,"15min":6.6349999999999998},"max":{"1min":35.213999999999999,"5min":35.213999999999999,"15min":336.55599999999998},"last":16.579000000000001},{"interface":"front","average":{"1min":10.366,"5min":11.09,"15min":19.364000000000001},"min":{"1min":7.0019999999999998,"5min":6.4859999999999998,"15min":6.4790000000000001},"max":{"1min":14.853,"5min":23.786000000000001,"15min":357.50900000000001},"last":8.2149999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":14.206,"5min":15.972,"15min":20.510999999999999},"min":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":8.1080000000000005},"max":{"1min":34.234000000000002,"5min":34.234000000000002,"15min":255.52500000000001},"last":9.6910000000000007},{"interface":"front","average":{"1min":19.222000000000001,"5min":15.115,"15min":19.218},"min":{"1min":15.247999999999999,"5min":7.7080000000000002,"15min":7.7080000000000002},"max":{"1min":37.884,"5min":37.884,"15min":312.791},"last":18.312000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":15.161,"5min":15.696,"15min":23.895},"min":{"1min":9.7590000000000003,"5min":9.6940000000000008,"15min":8.7330000000000005},"max":{"1min":23.195,"5min":23.195,"15min":413.48899999999998},"last":13.362},{"interface":"front","average":{"1min":17.57,"5min":16.561,"15min":21.308},"min":{"1min":9.4770000000000003,"5min":9.0809999999999995,"15min":6.6820000000000004},"max":{"1min":30.414999999999999,"5min":30.414999999999999,"15min":417.952},"last":15.034000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575241,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70972,"kb_used_data":7432,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300868,"statfs":{"total":96636764160,"available":96564088832,"internally_reserved":0,"allocated":7610368,"data_stored":4206728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":13.364000000000001,"5min":11.880000000000001,"15min":16.117000000000001},"min":{"1min":8.8580000000000005,"5min":5.2729999999999997,"15min":4.3360000000000003},"max":{"1min":39.423000000000002,"5min":39.423000000000002,"15min":203.19300000000001},"last":8.9830000000000005},{"interface":"front","average":{"1min":17.373999999999999,"5min":15.071999999999999,"15min":18.241},"min":{"1min":11.513,"5min":6.4269999999999996,"15min":4.4199999999999999},"max":{"1min":40.698999999999998,"5min":40.698999999999998,"15min":204.13},"last":14.613}]},{"osd":1,"last update":"Fri Dec 1 17:35:57 2023","interfaces":[{"interface":"back","average":{"1min":7.9080000000000004,"5min":8.2289999999999992,"15min":15.226000000000001},"min":{"1min":6.1139999999999999,"5min":5.2300000000000004,"15min":5.0369999999999999},"max":{"1min":10.85,"5min":17.564,"15min":167.82900000000001},"last":6.9969999999999999},{"interface":"front","average":{"1min":15.026,"5min":14.375999999999999,"15min":16.369},"min":{"1min":11.634,"5min":7.4870000000000001,"15min":5.484},"max":{"1min":22.280999999999999,"5min":22.280999999999999,"15min":154.137},"last":12.805}]},{"osd":2,"last update":"Fri Dec 1 17:36:23 2023","interfaces":[{"interface":"back","average":{"1min":11.753,"5min":10.013999999999999,"15min":16.608000000000001},"min":{"1min":5.8700000000000001,"5min":5.1820000000000004,"15min":5.1820000000000004},"max":{"1min":26.274999999999999,"5min":26.274999999999999,"15min":230.40600000000001},"last":13.621},{"interface":"front","average":{"1min":14.275,"5min":12.394,"15min":17.012},"min":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"max":{"1min":29.861000000000001,"5min":29.861000000000001,"15min":268.68000000000001},"last":9.5510000000000002}]},{"osd":3,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":13.083,"15min":20.297000000000001},"min":{"1min":8.1359999999999992,"5min":6.5110000000000001,"15min":6.5110000000000001},"max":{"1min":85.724000000000004,"5min":85.724000000000004,"15min":290.15300000000002},"last":8.1999999999999993},{"interface":"front","average":{"1min":14.601000000000001,"5min":11.683,"15min":20.149000000000001},"min":{"1min":6.6319999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"max":{"1min":83.183000000000007,"5min":83.183000000000007,"15min":305.41399999999999},"last":6.1669999999999998}]},{"osd":4,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":14.577,"5min":14.051,"15min":18.489000000000001},"min":{"1min":10.007999999999999,"5min":8.5489999999999995,"15min":6.9299999999999997},"max":{"1min":17.609999999999999,"5min":22.071000000000002,"15min":149.93000000000001},"last":11.746},{"interface":"front","average":{"1min":17.727,"5min":17.503,"15min":23.488},"min":{"1min":14.632,"5min":11.496,"15min":9.7810000000000006},"max":{"1min":26.896999999999998,"5min":27.422000000000001,"15min":239.37},"last":16.341000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":17.140000000000001,"5min":13.266,"15min":20.885000000000002},"min":{"1min":10.646000000000001,"5min":7.3209999999999997,"15min":6.8460000000000001},"max":{"1min":26.053999999999998,"5min":26.053999999999998,"15min":282.31200000000001},"last":17.155000000000001},{"interface":"front","average":{"1min":9.9749999999999996,"5min":13.909000000000001,"15min":19.908999999999999},"min":{"1min":6.8929999999999998,"5min":6.1529999999999996,"15min":6.1529999999999996},"max":{"1min":17.657,"5min":20.282,"15min":193.83600000000001},"last":11.114000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":18.216999999999999,"5min":17.367999999999999,"15min":20.030999999999999},"min":{"1min":13.897,"5min":11.176,"15min":6.625},"max":{"1min":24.832999999999998,"5min":24.832999999999998,"15min":209.25},"last":15.367000000000001},{"interface":"front","average":{"1min":19.565999999999999,"5min":19.302,"15min":24.709},"min":{"1min":15.249000000000001,"5min":14.266999999999999,"15min":8.9420000000000002},"max":{"1min":27.905000000000001,"5min":27.905000000000001,"15min":193.184},"last":17.757000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:36:28.982 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-12-01T17:36:28.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:36:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.088+0000 7fcc652b4640 10 monclient: get_monmap_and_config 2023-12-01T17:36:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.088+0000 7fcc652b4640 10 monclient: build_initial_monmap 2023-12-01T17:36:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.088+0000 7fcc652b4640 10 monclient: monmap: 2023-12-01T17:36:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:29.089648+0000 2023-12-01T17:36:29.088 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:29.089648+0000 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:29.089 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.089+0000 7fcc652b4640 1 Processor -- start 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.089+0000 7fcc652b4640 1 -- start start 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.089+0000 7fcc652b4640 10 monclient: init 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc600a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc6013f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient(hunting): picked mon.noname-b con 0x7fcc600a8e00 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:29.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc601423b0 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc600b21f0 con 0x7fcc600a8a20 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc60142f20 con 0x7fcc601423b0 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc600a9cc0 con 0x7fcc600a8e00 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc652b4640 10 monclient(hunting): authenticate will time out at 3320.327694s 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc5e7fc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc6013f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc5effd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fcc600a8a20 0x7fcc6013eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49332/0 (socket says 172.21.15.67:49332) 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.090+0000 7fcc5effd640 1 -- 172.21.15.67:0/3107241626 learned_addr learned my addr 172.21.15.67:0/3107241626 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.091+0000 7fcc5e7fc640 10 monclient(hunting): get_auth_request con 0x7fcc600a8e00 auth_method 0 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.091+0000 7fcc5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.091+0000 7fcc5e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.091+0000 7fcc5e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.091+0000 7fcc5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.092 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.091+0000 7fcc5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.091+0000 7fcc5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.093+0000 7fcc5dffb640 1 -- 172.21.15.67:0/3107241626 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097899366 0 0) 0x7fcc60142f20 con 0x7fcc601423b0 2023-12-01T17:36:29.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.093+0000 7fcc5dffb640 10 monclient(hunting): _negotiate 2023-12-01T17:36:29.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.093+0000 7fcc5dffb640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.093+0000 7fcc5dffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.093+0000 7fcc5dffb640 10 monclient(hunting): my global_id is 23143 2023-12-01T17:36:29.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.093+0000 7fcc5dffb640 1 -- 172.21.15.67:0/3107241626 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c003520 con 0x7fcc601423b0 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 10 monclient(hunting): handle_auth_done global_id 17903 payload 274 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 1 -- 172.21.15.67:0/3107241626 >> v1:172.21.15.92:6789/0 conn(0x7fcc601423b0 legacy=0x7fcc60142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 1 -- 172.21.15.67:0/3107241626 >> v1:172.21.15.67:6789/0 conn(0x7fcc600a8a20 legacy=0x7fcc6013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 10 monclient: _finish_hunting 0 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 1 monclient: found mon.noname-b 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5dffb640 1 -- 172.21.15.67:0/3107241626 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275215848 0 0) 0x7fcc600b21f0 con 0x7fcc600a8a20 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5e7fc640 1 -- 172.21.15.67:0/3107241626 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc600a9ed0 con 0x7fcc600a8e00 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.094+0000 7fcc5dffb640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-12-01T17:36:29.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.096+0000 7fcc5e7fc640 1 --2- 172.21.15.67:0/3107241626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc6013f5c0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fcc480097d0 tx=0x7fcc4802b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b1c0556ece053b server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 1 -- 172.21.15.67:0/3107241626 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48031770 con 0x7fcc600a8e00 2023-12-01T17:36:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 10 monclient: dump: 2023-12-01T17:36:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.099 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc5dffb640 1 -- 172.21.15.67:0/3107241626 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc480318d0 con 0x7fcc600a8e00 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.099+0000 7fcc652b4640 5 monclient: authenticate success, global_id 17903 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc5dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:29.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 4 set_mon_vals no callback set 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc5dffb640 1 -- 172.21.15.67:0/3107241626 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc48031c60 con 0x7fcc600a8e00 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc5dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc5dffb640 10 monclient: dump: 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.101 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 10 monclient: shutdown 2023-12-01T17:36:29.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:29.104 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 1 -- 172.21.15.67:0/3107241626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 msgr2=0x7fcc6013f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 1 --2- 172.21.15.67:0/3107241626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc6013f5c0 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7fcc480097d0 tx=0x7fcc4802b810 comp rx=0 tx=0).stop 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 1 -- 172.21.15.67:0/3107241626 shutdown_connections 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 1 --2- 172.21.15.67:0/3107241626 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc6013f5c0 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 1 -- 172.21.15.67:0/3107241626 >> 172.21.15.67:0/3107241626 conn(0x7fcc600b7870 msgr2=0x7fcc600b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 1 -- 172.21.15.67:0/3107241626 shutdown_connections 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.100+0000 7fcc652b4640 1 -- 172.21.15.67:0/3107241626 wait complete. 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.101+0000 7fcc652b4640 10 monclient: build_initial_monmap 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.101+0000 7fcc652b4640 10 monclient: monmap: 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:29.105 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.101+0000 7fcc652b4640 1 librados: starting msgr at 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.101+0000 7fcc652b4640 1 librados: starting objecter 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 Processor -- start 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 -- start start 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 librados: setting wanted keys 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 librados: calling monclient init 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient: init 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:36:29.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc600a8a20 0x7fcc60134f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): picked mon.noname-a con 0x7fcc600a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc601346b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): picked mon.noname-c con 0x7fcc600a8e00 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc601423b0 0x7fcc601da460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): picked mon.noname-b con 0x7fcc601423b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc601373c0 con 0x7fcc600a8a20 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc60134bf0 con 0x7fcc601423b0 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc60136ec0 con 0x7fcc600a8e00 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.102+0000 7fcc652b4640 10 monclient(hunting): authenticate will time out at 3320.340084s 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc601423b0 0x7fcc601da460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5e7fc640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc601346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc601423b0 0x7fcc601da460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47690/0 (socket says 172.21.15.67:47690) 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 1 -- 172.21.15.67:0/2286228621 learned_addr learned my addr 172.21.15.67:0/2286228621 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 10 monclient(hunting): get_auth_request con 0x7fcc601423b0 auth_method 0 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5effd640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc600a8a20 0x7fcc60134f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5e7fc640 10 monclient(hunting): get_auth_request con 0x7fcc600a8e00 auth_method 0 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5effd640 10 monclient(hunting): get_auth_request con 0x7fcc600a8a20 auth_method 0 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5effd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.103+0000 7fcc5f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.104+0000 7fcc5e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.104+0000 7fcc5e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.104+0000 7fcc5e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.104+0000 7fcc5effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.104+0000 7fcc5effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.104+0000 7fcc5effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 10 monclient(hunting): handle_auth_done global_id 23146 payload 824 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 1 -- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 msgr2=0x7fcc601346b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc601346b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 1 -- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc600a8a20 msgr2=0x7fcc60134f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc600a8a20 0x7fcc60134f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 10 monclient: _finish_hunting 0 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 1 monclient: found mon.noname-b 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.106+0000 7fcc5f7fe640 1 -- 172.21.15.67:0/2286228621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc48009400 con 0x7fcc601423b0 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.108+0000 7fcc5f7fe640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc601423b0 0x7fcc601da460 secure :-1 s=READY pgs=5881 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000c6e0 tx=0x7fcc5000cba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:29.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.108+0000 7fcc577fe640 10 client.?.objecter ms_handle_connect 0x7fcc601423b0 2023-12-01T17:36:29.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.108+0000 7fcc577fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 1 -- 172.21.15.67:0/2286228621 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc50003b70 con 0x7fcc601423b0 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 10 monclient: dump: 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.112 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 10 monclient: _finish_auth 0 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 10 monclient: _check_auth_tickets 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 5 monclient: authenticate success, global_id 23146 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.111+0000 7fcc577fe640 1 -- 172.21.15.67:0/2286228621 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc50003cd0 con 0x7fcc601423b0 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc577fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc577fe640 1 -- 172.21.15.67:0/2286228621 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc500052d0 con 0x7fcc601423b0 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc577fe640 10 monclient: dump: 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.113 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc5d7fa640 4 set_mon_vals no callback set 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 monclient: _renew_subs 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc601da9a0 con 0x7fcc601423b0 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 client.23146.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 monclient: _renew_subs 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc601dae30 con 0x7fcc601423b0 2023-12-01T17:36:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 1 librados: init done 2023-12-01T17:36:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.112+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc60136910 con 0x7fcc601423b0 2023-12-01T17:36:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.351+0000 7fcc577fe640 1 -- 172.21.15.67:0/2286228621 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fcc50005430 con 0x7fcc601423b0 2023-12-01T17:36:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.351+0000 7fcc577fe640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fcc4006c8a0 0x7fcc4006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.351+0000 7fcc5effd640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fcc4006c8a0 0x7fcc4006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.351+0000 7fcc5effd640 10 monclient: get_auth_request con 0x7fcc4006c8a0 auth_method 0 2023-12-01T17:36:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.352+0000 7fcc5effd640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fcc4006c8a0 0x7fcc4006ed60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcc58000f60 tx=0x7fcc58007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:29.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.352+0000 7fcc577fe640 10 client.23146.objecter ms_handle_connect 0x7fcc4006c8a0 2023-12-01T17:36:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.353+0000 7fcc577fe640 1 -- 172.21.15.67:0/2286228621 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1006..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fcc50058b70 con 0x7fcc601423b0 2023-12-01T17:36:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.353+0000 7fcc577fe640 10 client.23146.objecter ms_dispatch 0x7fcc60000b90 osd_map(1099..1099 src has 1006..1099) v4 2023-12-01T17:36:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.353+0000 7fcc577fe640 3 client.23146.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.353+0000 7fcc577fe640 3 client.23146.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:29.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.353+0000 7fcc577fe640 20 client.23146.objecter dump_active .. 0 homeless 2023-12-01T17:36:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.363+0000 7fcc577fe640 1 -- 172.21.15.67:0/2286228621 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc50054eb0 con 0x7fcc601423b0 2023-12-01T17:36:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.363+0000 7fcc577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.363+0000 7fcc577fe640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:29.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.548+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcc60134170 con 0x7fcc4006c8a0 2023-12-01T17:36:29.554 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.554+0000 7fcc577fe640 1 -- 172.21.15.67:0/2286228621 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151451 (secure 0 0 0) 0x7fcc60134170 con 0x7fcc4006c8a0 2023-12-01T17:36:29.555 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:36:29.556 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:36:29.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 10 librados: watch_flush enter 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 10 librados: watch_flush exit 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 20 client.23146.objecter shutdown clearing up homeless session... 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 10 client.23146.objecter successfully canceled tick 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fcc4006c8a0 msgr2=0x7fcc4006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fcc4006c8a0 0x7fcc4006ed60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fcc58000f60 tx=0x7fcc58007d50 comp rx=0 tx=0).stop 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 10 monclient: shutdown 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc601423b0 msgr2=0x7fcc601da460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc601423b0 0x7fcc601da460 secure :-1 s=READY pgs=5881 cs=0 l=1 rev1=1 crypto rx=0x7fcc5000c6e0 tx=0x7fcc5000cba0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 shutdown_connections 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcc601423b0 0x7fcc601da460 unknown :-1 s=CLOSED pgs=5881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fcc4006c8a0 0x7fcc4006ed60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fcc600a8e00 0x7fcc601346b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 --2- 172.21.15.67:0/2286228621 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc600a8a20 0x7fcc60134f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.559 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 >> 172.21.15.67:0/2286228621 conn(0x7fcc600b7870 msgr2=0x7fcc600b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:29.559 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 shutdown_connections 2023-12-01T17:36:29.559 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 -- 172.21.15.67:0/2286228621 wait complete. 2023-12-01T17:36:29.559 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.557+0000 7fcc652b4640 1 librados: shutdown 2023-12-01T17:36:29.567 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":2081,"stamp":"2023-12-01T17:36:28.773857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":575752,"kb_used_data":59100,"kb_used_omap":12,"kb_used_meta":513139,"kb_avail":754398968,"statfs":{"total":773094113280,"available":772504543232,"internally_reserved":0,"allocated":60518400,"data_stored":33737203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":525454896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":409,"apply_latency_ms":409,"commit_latency_ns":409000000,"apply_latency_ns":409000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000700"},"pg_stats":[{"pgid":"22.6","version":"0'0","reported_seq":1045,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.677997+0000","last_change":"2023-12-01T17:05:14.287690+0000","last_active":"2023-12-01T17:36:22.677997+0000","last_peered":"2023-12-01T17:36:22.677997+0000","last_clean":"2023-12-01T17:36:22.677997+0000","last_became_active":"2023-12-01T17:05:14.257152+0000","last_became_peered":"2023-12-01T17:05:14.257152+0000","last_unstale":"2023-12-01T17:36:22.677997+0000","last_undegraded":"2023-12-01T17:36:22.677997+0000","last_fullsized":"2023-12-01T17:36:22.677997+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:59:48.591916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1053,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.536581+0000","last_change":"2023-12-01T17:05:21.303515+0000","last_active":"2023-12-01T17:36:21.536581+0000","last_peered":"2023-12-01T17:36:21.536581+0000","last_clean":"2023-12-01T17:36:21.536581+0000","last_became_active":"2023-12-01T17:05:21.296791+0000","last_became_peered":"2023-12-01T17:05:21.296791+0000","last_unstale":"2023-12-01T17:36:21.536581+0000","last_undegraded":"2023-12-01T17:36:21.536581+0000","last_fullsized":"2023-12-01T17:36:21.536581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:51:36.558228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1056,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.138780+0000","last_change":"2023-12-01T17:05:13.270729+0000","last_active":"2023-12-01T17:36:26.138780+0000","last_peered":"2023-12-01T17:36:26.138780+0000","last_clean":"2023-12-01T17:36:26.138780+0000","last_became_active":"2023-12-01T17:05:13.265267+0000","last_became_peered":"2023-12-01T17:05:13.265267+0000","last_unstale":"2023-12-01T17:36:26.138780+0000","last_undegraded":"2023-12-01T17:36:26.138780+0000","last_fullsized":"2023-12-01T17:36:26.138780+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:57:02.423320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1049,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.596184+0000","last_change":"2023-12-01T17:05:38.668209+0000","last_active":"2023-12-01T17:36:22.596184+0000","last_peered":"2023-12-01T17:36:22.596184+0000","last_clean":"2023-12-01T17:36:22.596184+0000","last_became_active":"2023-12-01T17:05:38.651540+0000","last_became_peered":"2023-12-01T17:05:38.651540+0000","last_unstale":"2023-12-01T17:36:22.596184+0000","last_undegraded":"2023-12-01T17:36:22.596184+0000","last_fullsized":"2023-12-01T17:36:22.596184+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:43:47.235945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1058,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.693835+0000","last_change":"2023-12-01T17:05:45.108671+0000","last_active":"2023-12-01T17:36:21.693835+0000","last_peered":"2023-12-01T17:36:21.693835+0000","last_clean":"2023-12-01T17:36:21.693835+0000","last_became_active":"2023-12-01T17:05:45.102496+0000","last_became_peered":"2023-12-01T17:05:45.102496+0000","last_unstale":"2023-12-01T17:36:21.693835+0000","last_undegraded":"2023-12-01T17:36:21.693835+0000","last_fullsized":"2023-12-01T17:36:21.693835+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:35:01.639680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.590006+0000","last_change":"2023-12-01T17:05:21.134670+0000","last_active":"2023-12-01T17:36:22.590006+0000","last_peered":"2023-12-01T17:36:22.590006+0000","last_clean":"2023-12-01T17:36:22.590006+0000","last_became_active":"2023-12-01T17:05:21.112377+0000","last_became_peered":"2023-12-01T17:05:21.112377+0000","last_unstale":"2023-12-01T17:36:22.590006+0000","last_undegraded":"2023-12-01T17:36:22.590006+0000","last_fullsized":"2023-12-01T17:36:22.590006+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:40:39.378498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.131517+0000","last_change":"2023-12-01T17:05:12.338713+0000","last_active":"2023-12-01T17:36:26.131517+0000","last_peered":"2023-12-01T17:36:26.131517+0000","last_clean":"2023-12-01T17:36:26.131517+0000","last_became_active":"2023-12-01T17:05:12.325443+0000","last_became_peered":"2023-12-01T17:05:12.325443+0000","last_unstale":"2023-12-01T17:36:26.131517+0000","last_undegraded":"2023-12-01T17:36:26.131517+0000","last_fullsized":"2023-12-01T17:36:26.131517+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:16:52.693300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.577087+0000","last_change":"2023-12-01T17:05:35.977136+0000","last_active":"2023-12-01T17:36:22.577087+0000","last_peered":"2023-12-01T17:36:22.577087+0000","last_clean":"2023-12-01T17:36:22.577087+0000","last_became_active":"2023-12-01T17:05:35.971286+0000","last_became_peered":"2023-12-01T17:05:35.971286+0000","last_unstale":"2023-12-01T17:36:22.577087+0000","last_undegraded":"2023-12-01T17:36:22.577087+0000","last_fullsized":"2023-12-01T17:36:22.577087+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:40:27.793735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1046,"reported_epoch":1096,"state":"active+clean","last_fresh":"2023-12-01T17:36:25.149276+0000","last_change":"2023-12-01T17:05:33.458007+0000","last_active":"2023-12-01T17:36:25.149276+0000","last_peered":"2023-12-01T17:36:25.149276+0000","last_clean":"2023-12-01T17:36:25.149276+0000","last_became_active":"2023-12-01T17:05:33.448448+0000","last_became_peered":"2023-12-01T17:05:33.448448+0000","last_unstale":"2023-12-01T17:36:25.149276+0000","last_undegraded":"2023-12-01T17:36:25.149276+0000","last_fullsized":"2023-12-01T17:36:25.149276+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:03:13.718758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1051,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.606030+0000","last_change":"2023-12-01T17:05:36.335708+0000","last_active":"2023-12-01T17:36:22.606030+0000","last_peered":"2023-12-01T17:36:22.606030+0000","last_clean":"2023-12-01T17:36:22.606030+0000","last_became_active":"2023-12-01T17:05:36.329117+0000","last_became_peered":"2023-12-01T17:05:36.329117+0000","last_unstale":"2023-12-01T17:36:22.606030+0000","last_undegraded":"2023-12-01T17:36:22.606030+0000","last_fullsized":"2023-12-01T17:36:22.606030+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T00:46:41.192583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.140840+0000","last_change":"2023-12-01T17:05:11.457400+0000","last_active":"2023-12-01T17:36:26.140840+0000","last_peered":"2023-12-01T17:36:26.140840+0000","last_clean":"2023-12-01T17:36:26.140840+0000","last_became_active":"2023-12-01T17:05:11.452081+0000","last_became_peered":"2023-12-01T17:05:11.452081+0000","last_unstale":"2023-12-01T17:36:26.140840+0000","last_undegraded":"2023-12-01T17:36:26.140840+0000","last_fullsized":"2023-12-01T17:36:26.140840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:10:31.762918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1051,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.526103+0000","last_change":"2023-12-01T17:05:36.195013+0000","last_active":"2023-12-01T17:36:21.526103+0000","last_peered":"2023-12-01T17:36:21.526103+0000","last_clean":"2023-12-01T17:36:21.526103+0000","last_became_active":"2023-12-01T17:05:36.185378+0000","last_became_peered":"2023-12-01T17:05:36.185378+0000","last_unstale":"2023-12-01T17:36:21.526103+0000","last_undegraded":"2023-12-01T17:36:21.526103+0000","last_fullsized":"2023-12-01T17:36:21.526103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:38:11.192241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"22'76","reported_seq":1159,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.489678+0000","last_change":"2023-12-01T17:03:54.894752+0000","last_active":"2023-12-01T17:36:22.489678+0000","last_peered":"2023-12-01T17:36:22.489678+0000","last_clean":"2023-12-01T17:36:22.489678+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:36:22.489678+0000","last_undegraded":"2023-12-01T17:36:22.489678+0000","last_fullsized":"2023-12-01T17:36:22.489678+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:51.457677+0000","last_clean_scrub_stamp":"2023-12-01T17:03:51.457677+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:12:02.023301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1049,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.660940+0000","last_change":"2023-12-01T17:04:01.495183+0000","last_active":"2023-12-01T17:36:22.660940+0000","last_peered":"2023-12-01T17:36:22.660940+0000","last_clean":"2023-12-01T17:36:22.660940+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:36:22.660940+0000","last_undegraded":"2023-12-01T17:36:22.660940+0000","last_fullsized":"2023-12-01T17:36:22.660940+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:13:19.704857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1044,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.563641+0000","last_change":"2023-12-01T17:05:15.674793+0000","last_active":"2023-12-01T17:36:22.563641+0000","last_peered":"2023-12-01T17:36:22.563641+0000","last_clean":"2023-12-01T17:36:22.563641+0000","last_became_active":"2023-12-01T17:05:15.659631+0000","last_became_peered":"2023-12-01T17:05:15.659631+0000","last_unstale":"2023-12-01T17:36:22.563641+0000","last_undegraded":"2023-12-01T17:36:22.563641+0000","last_fullsized":"2023-12-01T17:36:22.563641+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:04:05.554372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1051,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.638318+0000","last_change":"2023-12-01T17:04:01.146575+0000","last_active":"2023-12-01T17:36:22.638318+0000","last_peered":"2023-12-01T17:36:22.638318+0000","last_clean":"2023-12-01T17:36:22.638318+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:36:22.638318+0000","last_undegraded":"2023-12-01T17:36:22.638318+0000","last_fullsized":"2023-12-01T17:36:22.638318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:18:50.459254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1047,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.513553+0000","last_change":"2023-12-01T17:05:11.335421+0000","last_active":"2023-12-01T17:36:22.513553+0000","last_peered":"2023-12-01T17:36:22.513553+0000","last_clean":"2023-12-01T17:36:22.513553+0000","last_became_active":"2023-12-01T17:05:11.329382+0000","last_became_peered":"2023-12-01T17:05:11.329382+0000","last_unstale":"2023-12-01T17:36:22.513553+0000","last_undegraded":"2023-12-01T17:36:22.513553+0000","last_fullsized":"2023-12-01T17:36:22.513553+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:01:09.667175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1057,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.528202+0000","last_change":"2023-12-01T17:04:01.107520+0000","last_active":"2023-12-01T17:36:21.528202+0000","last_peered":"2023-12-01T17:36:21.528202+0000","last_clean":"2023-12-01T17:36:21.528202+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:36:21.528202+0000","last_undegraded":"2023-12-01T17:36:21.528202+0000","last_fullsized":"2023-12-01T17:36:21.528202+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:47:47.456810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1053,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.143519+0000","last_change":"2023-12-01T17:05:16.493438+0000","last_active":"2023-12-01T17:36:26.143519+0000","last_peered":"2023-12-01T17:36:26.143519+0000","last_clean":"2023-12-01T17:36:26.143519+0000","last_became_active":"2023-12-01T17:05:16.481672+0000","last_became_peered":"2023-12-01T17:05:16.481672+0000","last_unstale":"2023-12-01T17:36:26.143519+0000","last_undegraded":"2023-12-01T17:36:26.143519+0000","last_fullsized":"2023-12-01T17:36:26.143519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:22:21.064006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1050,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.647330+0000","last_change":"2023-12-01T17:04:01.438358+0000","last_active":"2023-12-01T17:36:22.647330+0000","last_peered":"2023-12-01T17:36:22.647330+0000","last_clean":"2023-12-01T17:36:22.647330+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:36:22.647330+0000","last_undegraded":"2023-12-01T17:36:22.647330+0000","last_fullsized":"2023-12-01T17:36:22.647330+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:45:24.963538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1050,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.533853+0000","last_change":"2023-12-01T17:05:14.953876+0000","last_active":"2023-12-01T17:36:21.533853+0000","last_peered":"2023-12-01T17:36:21.533853+0000","last_clean":"2023-12-01T17:36:21.533853+0000","last_became_active":"2023-12-01T17:05:14.948506+0000","last_became_peered":"2023-12-01T17:05:14.948506+0000","last_unstale":"2023-12-01T17:36:21.533853+0000","last_undegraded":"2023-12-01T17:36:21.533853+0000","last_fullsized":"2023-12-01T17:36:21.533853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:52.139356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1060,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.137101+0000","last_change":"2023-12-01T17:04:01.467571+0000","last_active":"2023-12-01T17:36:26.137101+0000","last_peered":"2023-12-01T17:36:26.137101+0000","last_clean":"2023-12-01T17:36:26.137101+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:36:26.137101+0000","last_undegraded":"2023-12-01T17:36:26.137101+0000","last_fullsized":"2023-12-01T17:36:26.137101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:41:15.693504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1047,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.486427+0000","last_change":"2023-12-01T17:05:11.984847+0000","last_active":"2023-12-01T17:36:22.486427+0000","last_peered":"2023-12-01T17:36:22.486427+0000","last_clean":"2023-12-01T17:36:22.486427+0000","last_became_active":"2023-12-01T17:05:11.693016+0000","last_became_peered":"2023-12-01T17:05:11.693016+0000","last_unstale":"2023-12-01T17:36:22.486427+0000","last_undegraded":"2023-12-01T17:36:22.486427+0000","last_fullsized":"2023-12-01T17:36:22.486427+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:40:29.327298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1050,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.681781+0000","last_change":"2023-12-01T17:04:01.201853+0000","last_active":"2023-12-01T17:36:22.681781+0000","last_peered":"2023-12-01T17:36:22.681781+0000","last_clean":"2023-12-01T17:36:22.681781+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:36:22.681781+0000","last_undegraded":"2023-12-01T17:36:22.681781+0000","last_fullsized":"2023-12-01T17:36:22.681781+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:38:19.219620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1063,"reported_epoch":1095,"state":"active+clean","last_fresh":"2023-12-01T17:36:23.675866+0000","last_change":"2023-12-01T17:05:12.792097+0000","last_active":"2023-12-01T17:36:23.675866+0000","last_peered":"2023-12-01T17:36:23.675866+0000","last_clean":"2023-12-01T17:36:23.675866+0000","last_became_active":"2023-12-01T17:05:12.768446+0000","last_became_peered":"2023-12-01T17:05:12.768446+0000","last_unstale":"2023-12-01T17:36:23.675866+0000","last_undegraded":"2023-12-01T17:36:23.675866+0000","last_fullsized":"2023-12-01T17:36:23.675866+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:17:04.335014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1063,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.719744+0000","last_change":"2023-12-01T17:04:01.536883+0000","last_active":"2023-12-01T17:36:21.719744+0000","last_peered":"2023-12-01T17:36:21.719744+0000","last_clean":"2023-12-01T17:36:21.719744+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:36:21.719744+0000","last_undegraded":"2023-12-01T17:36:21.719744+0000","last_fullsized":"2023-12-01T17:36:21.719744+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:42:47.593805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1053,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.522414+0000","last_change":"2023-12-01T17:05:24.990862+0000","last_active":"2023-12-01T17:36:21.522414+0000","last_peered":"2023-12-01T17:36:21.522414+0000","last_clean":"2023-12-01T17:36:21.522414+0000","last_became_active":"2023-12-01T17:05:24.924547+0000","last_became_peered":"2023-12-01T17:05:24.924547+0000","last_unstale":"2023-12-01T17:36:21.522414+0000","last_undegraded":"2023-12-01T17:36:21.522414+0000","last_fullsized":"2023-12-01T17:36:21.522414+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:58.899609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.698718+0000","last_change":"2023-12-01T17:04:01.476480+0000","last_active":"2023-12-01T17:36:22.698718+0000","last_peered":"2023-12-01T17:36:22.698718+0000","last_clean":"2023-12-01T17:36:22.698718+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:36:22.698718+0000","last_undegraded":"2023-12-01T17:36:22.698718+0000","last_fullsized":"2023-12-01T17:36:22.698718+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:26:16.264231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1059,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.695845+0000","last_change":"2023-12-01T17:05:28.757137+0000","last_active":"2023-12-01T17:36:21.695845+0000","last_peered":"2023-12-01T17:36:21.695845+0000","last_clean":"2023-12-01T17:36:21.695845+0000","last_became_active":"2023-12-01T17:05:28.748541+0000","last_became_peered":"2023-12-01T17:05:28.748541+0000","last_unstale":"2023-12-01T17:36:21.695845+0000","last_undegraded":"2023-12-01T17:36:21.695845+0000","last_fullsized":"2023-12-01T17:36:21.695845+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:15:33.293632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1","version":"62'7","reported_seq":1064,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.578985+0000","last_change":"2023-12-01T17:05:38.783874+0000","last_active":"2023-12-01T17:36:22.578985+0000","last_peered":"2023-12-01T17:36:22.578985+0000","last_clean":"2023-12-01T17:36:22.578985+0000","last_became_active":"2023-12-01T17:05:38.775069+0000","last_became_peered":"2023-12-01T17:05:38.775069+0000","last_unstale":"2023-12-01T17:36:22.578985+0000","last_undegraded":"2023-12-01T17:36:22.578985+0000","last_fullsized":"2023-12-01T17:36:22.578985+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:09:31.642602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1052,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.565789+0000","last_change":"2023-12-01T17:05:17.531869+0000","last_active":"2023-12-01T17:36:22.565789+0000","last_peered":"2023-12-01T17:36:22.565789+0000","last_clean":"2023-12-01T17:36:22.565789+0000","last_became_active":"2023-12-01T17:05:17.447969+0000","last_became_peered":"2023-12-01T17:05:17.447969+0000","last_unstale":"2023-12-01T17:36:22.565789+0000","last_undegraded":"2023-12-01T17:36:22.565789+0000","last_fullsized":"2023-12-01T17:36:22.565789+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:20:05.487052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1048,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.496765+0000","last_change":"2023-12-01T17:05:12.380756+0000","last_active":"2023-12-01T17:36:22.496765+0000","last_peered":"2023-12-01T17:36:22.496765+0000","last_clean":"2023-12-01T17:36:22.496765+0000","last_became_active":"2023-12-01T17:05:12.334114+0000","last_became_peered":"2023-12-01T17:05:12.334114+0000","last_unstale":"2023-12-01T17:36:22.496765+0000","last_undegraded":"2023-12-01T17:36:22.496765+0000","last_fullsized":"2023-12-01T17:36:22.496765+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T00:52:19.170433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1065,"reported_epoch":1095,"state":"active+clean","last_fresh":"2023-12-01T17:36:23.690252+0000","last_change":"2023-12-01T17:05:39.346000+0000","last_active":"2023-12-01T17:36:23.690252+0000","last_peered":"2023-12-01T17:36:23.690252+0000","last_clean":"2023-12-01T17:36:23.690252+0000","last_became_active":"2023-12-01T17:05:39.340484+0000","last_became_peered":"2023-12-01T17:05:39.340484+0000","last_unstale":"2023-12-01T17:36:23.690252+0000","last_undegraded":"2023-12-01T17:36:23.690252+0000","last_fullsized":"2023-12-01T17:36:23.690252+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:50:46.835583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.517739+0000","last_change":"2023-12-01T17:05:58.777840+0000","last_active":"2023-12-01T17:36:22.517739+0000","last_peered":"2023-12-01T17:36:22.517739+0000","last_clean":"2023-12-01T17:36:22.517739+0000","last_became_active":"2023-12-01T17:05:58.772230+0000","last_became_peered":"2023-12-01T17:05:58.772230+0000","last_unstale":"2023-12-01T17:36:22.517739+0000","last_undegraded":"2023-12-01T17:36:22.517739+0000","last_fullsized":"2023-12-01T17:36:22.517739+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:05:58.781340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1044,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.667139+0000","last_change":"2023-12-01T17:05:11.225296+0000","last_active":"2023-12-01T17:36:22.667139+0000","last_peered":"2023-12-01T17:36:22.667139+0000","last_clean":"2023-12-01T17:36:22.667139+0000","last_became_active":"2023-12-01T17:05:11.209283+0000","last_became_peered":"2023-12-01T17:05:11.209283+0000","last_unstale":"2023-12-01T17:36:22.667139+0000","last_undegraded":"2023-12-01T17:36:22.667139+0000","last_fullsized":"2023-12-01T17:36:22.667139+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:39:18.388303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.566053+0000","last_change":"2023-12-01T17:05:17.271434+0000","last_active":"2023-12-01T17:36:22.566053+0000","last_peered":"2023-12-01T17:36:22.566053+0000","last_clean":"2023-12-01T17:36:22.566053+0000","last_became_active":"2023-12-01T17:05:17.215461+0000","last_became_peered":"2023-12-01T17:05:17.215461+0000","last_unstale":"2023-12-01T17:36:22.566053+0000","last_undegraded":"2023-12-01T17:36:22.566053+0000","last_fullsized":"2023-12-01T17:36:22.566053+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:14:02.813273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1044,"reported_epoch":1096,"state":"active+clean","last_fresh":"2023-12-01T17:36:25.151563+0000","last_change":"2023-12-01T17:05:21.697504+0000","last_active":"2023-12-01T17:36:25.151563+0000","last_peered":"2023-12-01T17:36:25.151563+0000","last_clean":"2023-12-01T17:36:25.151563+0000","last_became_active":"2023-12-01T17:05:21.668214+0000","last_became_peered":"2023-12-01T17:05:21.668214+0000","last_unstale":"2023-12-01T17:36:25.151563+0000","last_undegraded":"2023-12-01T17:36:25.151563+0000","last_fullsized":"2023-12-01T17:36:25.151563+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:07:59.125552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1060,"reported_epoch":1093,"state":"active+clean","last_fresh":"2023-12-01T17:36:21.688026+0000","last_change":"2023-12-01T17:05:46.022392+0000","last_active":"2023-12-01T17:36:21.688026+0000","last_peered":"2023-12-01T17:36:21.688026+0000","last_clean":"2023-12-01T17:36:21.688026+0000","last_became_active":"2023-12-01T17:05:46.016610+0000","last_became_peered":"2023-12-01T17:05:46.016610+0000","last_unstale":"2023-12-01T17:36:21.688026+0000","last_undegraded":"2023-12-01T17:36:21.688026+0000","last_fullsized":"2023-12-01T17:36:21.688026+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:47:36.167822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.636608+0000","last_change":"2023-12-01T17:05:37.365267+0000","last_active":"2023-12-01T17:36:22.636608+0000","last_peered":"2023-12-01T17:36:22.636608+0000","last_clean":"2023-12-01T17:36:22.636608+0000","last_became_active":"2023-12-01T17:05:37.358444+0000","last_became_peered":"2023-12-01T17:05:37.358444+0000","last_unstale":"2023-12-01T17:36:22.636608+0000","last_undegraded":"2023-12-01T17:36:22.636608+0000","last_fullsized":"2023-12-01T17:36:22.636608+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:48:13.502545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1055,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.162048+0000","last_change":"2023-12-01T17:05:13.509587+0000","last_active":"2023-12-01T17:36:26.162048+0000","last_peered":"2023-12-01T17:36:26.162048+0000","last_clean":"2023-12-01T17:36:26.162048+0000","last_became_active":"2023-12-01T17:05:13.499129+0000","last_became_peered":"2023-12-01T17:05:13.499129+0000","last_unstale":"2023-12-01T17:36:26.162048+0000","last_undegraded":"2023-12-01T17:36:26.162048+0000","last_fullsized":"2023-12-01T17:36:26.162048+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:25:34.541978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1046,"reported_epoch":1094,"state":"active+clean","last_fresh":"2023-12-01T17:36:22.649060+0000","last_change":"2023-12-01T17:05:37.748101+0000","last_active":"2023-12-01T17:36:22.649060+0000","last_peered":"2023-12-01T17:36:22.649060+0000","last_clean":"2023-12-01T17:36:22.649060+0000","last_became_active":"2023-12-01T17:05:37.740514+0000","last_became_peered":"2023-12-01T17:05:37.740514+0000","last_unstale":"2023-12-01T17:36:22.649060+0000","last_undegraded":"2023-12-01T17:36:22.649060+0000","last_fullsized":"2023-12-01T17:36:22.649060+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T04:17:18.757175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444421,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":71768,"kb_used_data":8028,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300072,"statfs":{"total":96636764160,"available":96563273728,"internally_reserved":0,"allocated":8220672,"data_stored":4735665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":20.553999999999998,"5min":12.125999999999999,"15min":15.801},"min":{"1min":6.5620000000000003,"5min":4.5899999999999999,"15min":4.5899999999999999},"max":{"1min":127.741,"5min":127.741,"15min":150.279},"last":8.2080000000000002},{"interface":"front","average":{"1min":29.547000000000001,"5min":14.099,"15min":17.922999999999998},"min":{"1min":8.5419999999999998,"5min":4.7229999999999999,"15min":4.7229999999999999},"max":{"1min":184.30600000000001,"5min":184.30600000000001,"15min":184.30600000000001},"last":10.763}]},{"osd":2,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":18.321999999999999,"5min":12.057,"15min":16.654},"min":{"1min":6.5709999999999997,"5min":5.4610000000000003,"15min":5.4610000000000003},"max":{"1min":99.156000000000006,"5min":99.156000000000006,"15min":208.18199999999999},"last":8.8569999999999993},{"interface":"front","average":{"1min":34.377000000000002,"5min":16.986999999999998,"15min":19.702000000000002},"min":{"1min":7.9260000000000002,"5min":6.8769999999999998,"15min":6.2030000000000003},"max":{"1min":216.785,"5min":216.785,"15min":216.785},"last":12.122999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":28.09,"5min":16.707999999999998,"15min":21.498000000000001},"min":{"1min":8.9960000000000004,"5min":7.8550000000000004,"15min":7.8550000000000004},"max":{"1min":243.88999999999999,"5min":243.88999999999999,"15min":276.07400000000001},"last":16.044},{"interface":"front","average":{"1min":30.187000000000001,"5min":19.353999999999999,"15min":24.324000000000002},"min":{"1min":13.336,"5min":10.973000000000001,"15min":7.2649999999999997},"max":{"1min":239.40000000000001,"5min":239.40000000000001,"15min":252.43299999999999},"last":36.654000000000003}]},{"osd":4,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":31.789000000000001,"5min":16.448,"15min":17.77},"min":{"1min":8.0429999999999993,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":230.511,"5min":230.511,"15min":230.511},"last":11.507999999999999},{"interface":"front","average":{"1min":33.975000000000001,"5min":17.349,"15min":21.777999999999999},"min":{"1min":7.6379999999999999,"5min":7.3979999999999997,"15min":7.3979999999999997},"max":{"1min":241.28100000000001,"5min":241.28100000000001,"15min":241.28100000000001},"last":9.6859999999999999}]},{"osd":5,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":27.834,"5min":16.713999999999999,"15min":18.786000000000001},"min":{"1min":6.859,"5min":6.859,"15min":6.149},"max":{"1min":191.78700000000001,"5min":191.78700000000001,"15min":191.78700000000001},"last":16.759},{"interface":"front","average":{"1min":17.850999999999999,"5min":15.477,"15min":22.440999999999999},"min":{"1min":7.3769999999999998,"5min":7.3769999999999998,"15min":7.0270000000000001},"max":{"1min":111.208,"5min":111.208,"15min":311.13299999999998},"last":13.367000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":22.614999999999998,"5min":17.094999999999999,"15min":18.754999999999999},"min":{"1min":11.262,"5min":10.199,"15min":7.0960000000000001},"max":{"1min":94.787999999999997,"5min":94.787999999999997,"15min":99.623000000000005},"last":33.271000000000001},{"interface":"front","average":{"1min":26.064,"5min":17.611000000000001,"15min":21.722999999999999},"min":{"1min":8.6600000000000001,"5min":8.6080000000000005,"15min":7.9909999999999997},"max":{"1min":153.83799999999999,"5min":153.83799999999999,"15min":186.11000000000001},"last":31.146000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":35.716000000000001,"5min":20.347000000000001,"15min":24.768999999999998},"min":{"1min":12.414999999999999,"5min":9.4819999999999993,"15min":9.1150000000000002},"max":{"1min":207.64599999999999,"5min":207.64599999999999,"15min":207.64599999999999},"last":35.598999999999997},{"interface":"front","average":{"1min":23.535,"5min":17.969999999999999,"15min":21.334},"min":{"1min":11.978999999999999,"5min":9.4469999999999992,"15min":9.0069999999999997},"max":{"1min":74.777000000000001,"5min":74.777000000000001,"15min":148.89599999999999},"last":14.211}]}]},{"osd":3,"up_from":16,"seq":68719477125,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70992,"kb_used_data":7228,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300848,"statfs":{"total":96636764160,"available":96564068352,"internally_reserved":0,"allocated":7401472,"data_stored":4081800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":57,"apply_latency_ms":57,"commit_latency_ns":57000000,"apply_latency_ns":57000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":38.427,"5min":15.26,"15min":18.015000000000001},"min":{"1min":6.8440000000000003,"5min":5.5540000000000003,"15min":5.2539999999999996},"max":{"1min":180.16,"5min":180.16,"15min":181.94200000000001},"last":10.919},{"interface":"front","average":{"1min":28.507000000000001,"5min":13.943,"15min":18.838000000000001},"min":{"1min":4.915,"5min":4.915,"15min":4.915},"max":{"1min":107.5,"5min":107.5,"15min":427.36500000000001},"last":13.970000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":30.577999999999999,"5min":15.76,"15min":20.832999999999998},"min":{"1min":8.1419999999999995,"5min":7.585,"15min":6.4039999999999999},"max":{"1min":146.48099999999999,"5min":146.48099999999999,"15min":227.309},"last":7.3890000000000002},{"interface":"front","average":{"1min":33.158000000000001,"5min":15.33,"15min":19.213999999999999},"min":{"1min":8.2870000000000008,"5min":6.9850000000000003,"15min":5.827},"max":{"1min":223.179,"5min":223.179,"15min":421.40600000000001},"last":9.0570000000000004}]},{"osd":2,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":20.815999999999999,"5min":13.278,"15min":22.062000000000001},"min":{"1min":7.9829999999999997,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":86.969999999999999,"5min":86.969999999999999,"15min":451.97899999999998},"last":16.805},{"interface":"front","average":{"1min":29.763999999999999,"5min":15.894,"15min":23.613},"min":{"1min":6.8540000000000001,"5min":6.7190000000000003,"15min":6.5170000000000003},"max":{"1min":223.84,"5min":223.84,"15min":384.58699999999999},"last":14.712999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":33.475999999999999,"5min":17.899999999999999,"15min":21.863},"min":{"1min":9.6440000000000001,"5min":8.5640000000000001,"15min":6.5330000000000004},"max":{"1min":244.989,"5min":244.989,"15min":424.09500000000003},"last":20.486000000000001},{"interface":"front","average":{"1min":31.292000000000002,"5min":17.065000000000001,"15min":21.056000000000001},"min":{"1min":8.9700000000000006,"5min":7.9870000000000001,"15min":7.0049999999999999},"max":{"1min":159.78200000000001,"5min":159.78200000000001,"15min":202.70500000000001},"last":11.794}]},{"osd":5,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":32.497,"5min":19.154,"15min":25.114000000000001},"min":{"1min":10.481999999999999,"5min":9.7400000000000002,"15min":9.0860000000000003},"max":{"1min":228.96700000000001,"5min":228.96700000000001,"15min":424.80099999999999},"last":19.100999999999999},{"interface":"front","average":{"1min":23.297999999999998,"5min":15.457000000000001,"15min":21.076000000000001},"min":{"1min":8.0359999999999996,"5min":7.2850000000000001,"15min":7.1710000000000003},"max":{"1min":87.635000000000005,"5min":87.635000000000005,"15min":231.65199999999999},"last":18.146000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":27.949000000000002,"5min":17.577000000000002,"15min":21.687999999999999},"min":{"1min":8.2859999999999996,"5min":8.2859999999999996,"15min":7.9340000000000002},"max":{"1min":136.684,"5min":136.684,"15min":203.99100000000001},"last":10.311999999999999},{"interface":"front","average":{"1min":30.741,"5min":19.934000000000001,"15min":25.875},"min":{"1min":12.468999999999999,"5min":10.978,"15min":10.978},"max":{"1min":147.88200000000001,"5min":147.88200000000001,"15min":203.34100000000001},"last":13.345000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":25.036000000000001,"5min":16.992999999999999,"15min":22.780000000000001},"min":{"1min":8.7509999999999994,"5min":7.9589999999999996,"15min":7.0259999999999998},"max":{"1min":105.342,"5min":105.342,"15min":196.05699999999999},"last":22.114999999999998},{"interface":"front","average":{"1min":27.623000000000001,"5min":19.024999999999999,"15min":25.555},"min":{"1min":11.186,"5min":11.186,"15min":10.492000000000001},"max":{"1min":102.29000000000001,"5min":102.29000000000001,"15min":236.84700000000001},"last":23.364000000000001}]}]},{"osd":2,"up_from":16,"seq":68719477123,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70996,"kb_used_data":7232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300844,"statfs":{"total":96636764160,"available":96564064256,"internally_reserved":0,"allocated":7405568,"data_stored":4081805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.219000000000001,"5min":14.260999999999999,"15min":20.582999999999998},"min":{"1min":6.4989999999999997,"5min":4.7110000000000003,"15min":4.6059999999999999},"max":{"1min":169.971,"5min":169.971,"15min":231.76499999999999},"last":9.3409999999999993},{"interface":"front","average":{"1min":29.649999999999999,"5min":14.412000000000001,"15min":19.628},"min":{"1min":5.3049999999999997,"5min":5.1890000000000001,"15min":4.8810000000000002},"max":{"1min":150.74199999999999,"5min":150.74199999999999,"15min":222.62},"last":10.647}]},{"osd":1,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":23.131,"5min":12.741,"15min":20.030000000000001},"min":{"1min":6.4980000000000002,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":182.83600000000001,"5min":182.83600000000001,"15min":220.96700000000001},"last":8.6430000000000007},{"interface":"front","average":{"1min":27.039999999999999,"5min":16.943999999999999,"15min":23.343},"min":{"1min":8.6869999999999994,"5min":8.6869999999999994,"15min":7.8639999999999999},"max":{"1min":196.886,"5min":196.886,"15min":213.27799999999999},"last":12.291}]},{"osd":3,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":40.603999999999999,"5min":17.788,"15min":22.448},"min":{"1min":7.0609999999999999,"5min":7.0609999999999999,"15min":6.734},"max":{"1min":177.04900000000001,"5min":177.04900000000001,"15min":219.51300000000001},"last":13.166},{"interface":"front","average":{"1min":41.942999999999998,"5min":19.474,"15min":24.347000000000001},"min":{"1min":7.6740000000000004,"5min":7.6740000000000004,"15min":6.9900000000000002},"max":{"1min":204.721,"5min":204.721,"15min":204.721},"last":19.106999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.542000000000002,"5min":16.068000000000001,"15min":21.274000000000001},"min":{"1min":8.6310000000000002,"5min":7.2060000000000004,"15min":6.6799999999999997},"max":{"1min":180.369,"5min":180.369,"15min":251.745},"last":26.879000000000001},{"interface":"front","average":{"1min":33.243000000000002,"5min":18.972000000000001,"15min":24.882999999999999},"min":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":7.5720000000000001},"max":{"1min":170.637,"5min":170.637,"15min":287.32100000000003},"last":25.776}]},{"osd":5,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":37.953000000000003,"5min":18.436,"15min":24.466999999999999},"min":{"1min":8.5060000000000002,"5min":8.3300000000000001,"15min":8.0020000000000007},"max":{"1min":206.56700000000001,"5min":206.56700000000001,"15min":210.012},"last":11.662000000000001},{"interface":"front","average":{"1min":33.348999999999997,"5min":19.402000000000001,"15min":25.024000000000001},"min":{"1min":9.1980000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"max":{"1min":232.91900000000001,"5min":232.91900000000001,"15min":232.91900000000001},"last":14.550000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":35.573999999999998,"5min":19.463000000000001,"15min":23.882999999999999},"min":{"1min":8.8710000000000004,"5min":8.8710000000000004,"15min":8.1829999999999998},"max":{"1min":182.07400000000001,"5min":182.07400000000001,"15min":237.21899999999999},"last":17.206},{"interface":"front","average":{"1min":38.033999999999999,"5min":20.099,"15min":27.759},"min":{"1min":12.598000000000001,"5min":9.5960000000000001,"15min":9.5649999999999995},"max":{"1min":185.90299999999999,"5min":185.90299999999999,"15min":241.92699999999999},"last":17.835000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":19.201000000000001,"5min":16.905000000000001,"15min":24.222000000000001},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.5690000000000008},"max":{"1min":68.075000000000003,"5min":68.075000000000003,"15min":341.73000000000002},"last":15.349},{"interface":"front","average":{"1min":22.43,"5min":18.937999999999999,"15min":27.407},"min":{"1min":10.853,"5min":10.853,"15min":10.853},"max":{"1min":89.631,"5min":89.631,"15min":343.17399999999998},"last":16.285}]}]},{"osd":0,"up_from":15,"seq":64424509828,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70588,"kb_used_data":7252,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301252,"statfs":{"total":96636764160,"available":96564482048,"internally_reserved":0,"allocated":7426048,"data_stored":4096103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":67,"apply_latency_ms":67,"commit_latency_ns":67000000,"apply_latency_ns":67000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 17:36:15 2023","interfaces":[{"interface":"back","average":{"1min":11.167,"5min":10.766999999999999,"15min":17.079000000000001},"min":{"1min":5.5579999999999998,"5min":5.5579999999999998,"15min":4.3550000000000004},"max":{"1min":33.322000000000003,"5min":33.322000000000003,"15min":210.43299999999999},"last":66.680999999999997},{"interface":"front","average":{"1min":12.661,"5min":10.788,"15min":18.704999999999998},"min":{"1min":6.242,"5min":6.0750000000000002,"15min":5.2590000000000003},"max":{"1min":44.020000000000003,"5min":44.020000000000003,"15min":257.30500000000001},"last":92.177000000000007}]},{"osd":2,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.808,"5min":14.16,"15min":19.648},"min":{"1min":8.2010000000000005,"5min":7.4740000000000002,"15min":6.452},"max":{"1min":94.155000000000001,"5min":94.155000000000001,"15min":239.24600000000001},"last":94.155000000000001},{"interface":"front","average":{"1min":16.765000000000001,"5min":13.634,"15min":18.114999999999998},"min":{"1min":8.3179999999999996,"5min":6.8579999999999997,"15min":5.6520000000000001},"max":{"1min":86.099999999999994,"5min":86.099999999999994,"15min":246.28299999999999},"last":86.099999999999994}]},{"osd":3,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":12.295999999999999,"15min":19.577000000000002},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"max":{"1min":128.08099999999999,"5min":128.08099999999999,"15min":240.017},"last":53.909999999999997},{"interface":"front","average":{"1min":28.396000000000001,"5min":17.071999999999999,"15min":25.015999999999998},"min":{"1min":9.452,"5min":8.3499999999999996,"15min":7.2779999999999996},"max":{"1min":136.40600000000001,"5min":136.40600000000001,"15min":249.86099999999999},"last":93.147000000000006}]},{"osd":4,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":15.007999999999999,"5min":12.295,"15min":18.792999999999999},"min":{"1min":7.5780000000000003,"5min":6.6909999999999998,"15min":6.6909999999999998},"max":{"1min":77.429000000000002,"5min":77.429000000000002,"15min":241.292},"last":196.71799999999999},{"interface":"front","average":{"1min":19.893999999999998,"5min":16.827000000000002,"15min":22.696000000000002},"min":{"1min":11.875,"5min":9.3759999999999994,"15min":7.2290000000000001},"max":{"1min":91.792000000000002,"5min":91.792000000000002,"15min":236.65799999999999},"last":207.38}]},{"osd":5,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":16.934000000000001,"5min":14.622,"15min":24.321999999999999},"min":{"1min":9.2949999999999999,"5min":8.6929999999999996,"15min":7.9400000000000004},"max":{"1min":45.228000000000002,"5min":45.228000000000002,"15min":241.91200000000001},"last":55.027000000000001},{"interface":"front","average":{"1min":16.408000000000001,"5min":14.084,"15min":21.462},"min":{"1min":7.2779999999999996,"5min":7.2779999999999996,"15min":6.4279999999999999},"max":{"1min":66.477000000000004,"5min":66.477000000000004,"15min":247.185},"last":73.453000000000003}]},{"osd":6,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":15.752000000000001,"15min":23.873999999999999},"min":{"1min":9.8810000000000002,"5min":8.2829999999999995,"15min":7.923},"max":{"1min":50.194000000000003,"5min":50.194000000000003,"15min":244.93799999999999},"last":68.665999999999997},{"interface":"front","average":{"1min":16.581,"5min":15.106999999999999,"15min":22.677},"min":{"1min":9.2609999999999992,"5min":9.2110000000000003,"15min":7.734},"max":{"1min":42.387999999999998,"5min":42.387999999999998,"15min":247.80500000000001},"last":83.817999999999998}]},{"osd":7,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":18.469999999999999,"5min":17.091999999999999,"15min":24.052},"min":{"1min":11.074999999999999,"5min":10.281000000000001,"15min":9.0820000000000007},"max":{"1min":37.758000000000003,"5min":37.758000000000003,"15min":238.58099999999999},"last":199.01400000000001},{"interface":"front","average":{"1min":18.225000000000001,"5min":16.898,"15min":24.481000000000002},"min":{"1min":11.706,"5min":9.3140000000000001,"15min":9.3140000000000001},"max":{"1min":36.850000000000001,"5min":36.850000000000001,"15min":273.142},"last":91.361000000000004}]}]},{"osd":6,"up_from":14,"seq":60129542534,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70568,"kb_used_data":7232,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301272,"statfs":{"total":96636764160,"available":96564502528,"internally_reserved":0,"allocated":7405568,"data_stored":4081805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":10.34,"5min":9.7390000000000008,"15min":13.387},"min":{"1min":4.8339999999999996,"5min":4.577,"15min":4.0730000000000004},"max":{"1min":25.731999999999999,"5min":25.731999999999999,"15min":129.89699999999999},"last":6.7709999999999999},{"interface":"front","average":{"1min":11.954000000000001,"5min":10.832000000000001,"15min":15.518000000000001},"min":{"1min":6.5010000000000003,"5min":6.4669999999999996,"15min":4.8129999999999997},"max":{"1min":34.526000000000003,"5min":34.526000000000003,"15min":155.99100000000001},"last":12.582000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":14.457000000000001,"5min":12.138,"15min":15.375},"min":{"1min":7.9969999999999999,"5min":6.702,"15min":6.2789999999999999},"max":{"1min":50.130000000000003,"5min":50.130000000000003,"15min":175.55000000000001},"last":10.417},{"interface":"front","average":{"1min":18.449000000000002,"5min":15.638999999999999,"15min":19.158000000000001},"min":{"1min":10.378,"5min":10.378,"15min":7.0490000000000004},"max":{"1min":54.060000000000002,"5min":54.060000000000002,"15min":130.91999999999999},"last":11.733000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":21.776,"5min":15.327,"15min":19.893999999999998},"min":{"1min":7.0010000000000003,"5min":6.8879999999999999,"15min":6.8879999999999999},"max":{"1min":172.459,"5min":172.459,"15min":191.71299999999999},"last":13.83},{"interface":"front","average":{"1min":19.948,"5min":11.276999999999999,"15min":16.713000000000001},"min":{"1min":4.4370000000000003,"5min":4.4370000000000003,"15min":4.4370000000000003},"max":{"1min":174.608,"5min":174.608,"15min":244.46299999999999},"last":5.1449999999999996}]},{"osd":3,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":19.504999999999999,"5min":15.193,"15min":20.762},"min":{"1min":10.653,"5min":8.2929999999999993,"15min":6.5830000000000002},"max":{"1min":70.066999999999993,"5min":70.066999999999993,"15min":223.97800000000001},"last":16.021999999999998},{"interface":"front","average":{"1min":18.291,"5min":14.847,"15min":22.268000000000001},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":6.8559999999999999},"max":{"1min":71.659999999999997,"5min":71.659999999999997,"15min":212.309},"last":17.539999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.582000000000001,"5min":14.68,"15min":19.289999999999999},"min":{"1min":10.567,"5min":8.952,"15min":7.0510000000000002},"max":{"1min":21.972000000000001,"5min":22.463000000000001,"15min":134.95699999999999},"last":15.204000000000001},{"interface":"front","average":{"1min":17.640999999999998,"5min":17.771000000000001,"15min":22.725000000000001},"min":{"1min":11.27,"5min":11.27,"15min":9.9809999999999999},"max":{"1min":22.663,"5min":22.663,"15min":135.798},"last":19.507000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.782,"5min":15.356,"15min":20.988},"min":{"1min":7.8040000000000003,"5min":7.8040000000000003,"15min":7.2869999999999999},"max":{"1min":20.332000000000001,"5min":20.966000000000001,"15min":288.577},"last":13.218999999999999},{"interface":"front","average":{"1min":17.925999999999998,"5min":17.777000000000001,"15min":23.446999999999999},"min":{"1min":12.861000000000001,"5min":12.124000000000001,"15min":10.234},"max":{"1min":23.948,"5min":23.948,"15min":142.77699999999999},"last":18.559999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":11.468999999999999,"5min":10.596,"15min":15.210000000000001},"min":{"1min":7.2649999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"max":{"1min":21.209,"5min":21.209,"15min":168.59700000000001},"last":9.2189999999999994},{"interface":"front","average":{"1min":17.047000000000001,"5min":16.718,"15min":21.393000000000001},"min":{"1min":11.954000000000001,"5min":8.9459999999999997,"15min":8.4770000000000003},"max":{"1min":24.709,"5min":25.518000000000001,"15min":183.054},"last":16.818000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542535,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74332,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297508,"statfs":{"total":96636764160,"available":96560648192,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":72,"apply_latency_ms":72,"commit_latency_ns":72000000,"apply_latency_ns":72000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":24.530999999999999,"5min":11.712,"15min":15.721},"min":{"1min":4.3620000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"max":{"1min":201.333,"5min":201.333,"15min":228.261},"last":6.4809999999999999},{"interface":"front","average":{"1min":27.792000000000002,"5min":13.209,"15min":17.690999999999999},"min":{"1min":6.5709999999999997,"5min":5.0339999999999998,"15min":4.6580000000000004},"max":{"1min":221.876,"5min":221.876,"15min":222.441},"last":16.992000000000001}]},{"osd":1,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.556000000000001,"5min":12.199999999999999,"15min":19.036000000000001},"min":{"1min":7.0490000000000004,"5min":6.0640000000000001,"15min":4.8899999999999997},"max":{"1min":66.040999999999997,"5min":66.040999999999997,"15min":216.20099999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":17.574000000000002,"5min":12.794,"15min":19.777000000000001},"min":{"1min":7.7430000000000003,"5min":6.5650000000000004,"15min":5.6210000000000004},"max":{"1min":65.156999999999996,"5min":65.156999999999996,"15min":255.95599999999999},"last":15.778}]},{"osd":2,"last update":"Fri Dec 1 17:35:58 2023","interfaces":[{"interface":"back","average":{"1min":12.641999999999999,"5min":11.792,"15min":17.571999999999999},"min":{"1min":6.468,"5min":6.468,"15min":6.468},"max":{"1min":39.947000000000003,"5min":39.947000000000003,"15min":179.81999999999999},"last":7.2530000000000001},{"interface":"front","average":{"1min":16.623999999999999,"5min":15.505000000000001,"15min":22.356999999999999},"min":{"1min":8.7070000000000007,"5min":8.7070000000000007,"15min":8.7070000000000007},"max":{"1min":42.759,"5min":42.759,"15min":224.47300000000001},"last":11.065}]},{"osd":3,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":13.112,"15min":19.452999999999999},"min":{"1min":6.819,"5min":6.819,"15min":5.9020000000000001},"max":{"1min":61.933,"5min":61.933,"15min":248.07300000000001},"last":19.225999999999999},{"interface":"front","average":{"1min":22.122,"5min":16.693000000000001,"15min":23.396000000000001},"min":{"1min":11.053000000000001,"5min":10.221,"15min":9.7989999999999995},"max":{"1min":62.886000000000003,"5min":62.886000000000003,"15min":235.02699999999999},"last":18.350999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":17.068999999999999,"5min":15.241,"15min":19.116},"min":{"1min":10.965999999999999,"5min":9.4930000000000003,"15min":8.1699999999999999},"max":{"1min":59.747,"5min":59.747,"15min":211.09399999999999},"last":9.8529999999999998},{"interface":"front","average":{"1min":14.052,"5min":14.444000000000001,"15min":18.030000000000001},"min":{"1min":9.6010000000000009,"5min":8.6850000000000005,"15min":7.0540000000000003},"max":{"1min":29.402000000000001,"5min":29.402000000000001,"15min":216.85300000000001},"last":12.993}]},{"osd":6,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":20.969999999999999,"5min":17.178000000000001,"15min":21.32},"min":{"1min":11.702,"5min":10.103,"15min":8.4339999999999993},"max":{"1min":111.396,"5min":111.396,"15min":223.89500000000001},"last":16.402999999999999},{"interface":"front","average":{"1min":16.765999999999998,"5min":14.01,"15min":14.154999999999999},"min":{"1min":7.5949999999999998,"5min":5.6829999999999998,"15min":5.6479999999999997},"max":{"1min":34.613,"5min":34.613,"15min":159.84399999999999},"last":13.773}]},{"osd":7,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":18.885999999999999,"5min":17.652000000000001,"15min":21.867000000000001},"min":{"1min":10.467000000000001,"5min":10.467000000000001,"15min":9.6430000000000007},"max":{"1min":62.256999999999998,"5min":62.256999999999998,"15min":226.06100000000001},"last":14.355},{"interface":"front","average":{"1min":21.974,"5min":17.885000000000002,"15min":21.588999999999999},"min":{"1min":9.3369999999999997,"5min":9.3369999999999997,"15min":8.1820000000000004},"max":{"1min":125.431,"5min":125.431,"15min":317.267},"last":12.250999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542535,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75536,"kb_used_data":7636,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94296304,"statfs":{"total":96636764160,"available":96559415296,"internally_reserved":0,"allocated":7819264,"data_stored":4337825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":7.8300000000000001,"5min":9.3949999999999996,"15min":16.763000000000002},"min":{"1min":4.1109999999999998,"5min":4.1109999999999998,"15min":3.7690000000000001},"max":{"1min":15.778,"5min":24.963999999999999,"15min":521.55899999999997},"last":4.609},{"interface":"front","average":{"1min":9.3529999999999998,"5min":9.8160000000000007,"15min":15.084},"min":{"1min":6.3360000000000003,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":13.864000000000001,"5min":21.491,"15min":281.125},"last":9.0220000000000002}]},{"osd":1,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":11.353999999999999,"5min":11.315,"15min":19.023},"min":{"1min":8.2140000000000004,"5min":6.7969999999999997,"15min":6.2770000000000001},"max":{"1min":14.935,"5min":21.539000000000001,"15min":308.99299999999999},"last":10.351000000000001},{"interface":"front","average":{"1min":16.753,"5min":15.666,"15min":22.716000000000001},"min":{"1min":10.832000000000001,"5min":9.1069999999999993,"15min":6.3079999999999998},"max":{"1min":31.140000000000001,"5min":31.140000000000001,"15min":289.79399999999998},"last":11.068}]},{"osd":2,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":16.849,"5min":15.061999999999999,"15min":21.503},"min":{"1min":11.24,"5min":9.1029999999999998,"15min":5.976},"max":{"1min":27.797999999999998,"5min":27.797999999999998,"15min":298.18200000000002},"last":14.058999999999999},{"interface":"front","average":{"1min":12.192,"5min":10.987,"15min":16.457999999999998},"min":{"1min":8.0640000000000001,"5min":6.4820000000000002,"15min":6.4820000000000002},"max":{"1min":25.771999999999998,"5min":25.771999999999998,"15min":223.768},"last":11.661}]},{"osd":3,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":12.680999999999999,"15min":18.734000000000002},"min":{"1min":8.6850000000000005,"5min":7.1580000000000004,"15min":6.2169999999999996},"max":{"1min":26.462,"5min":26.462,"15min":243.40299999999999},"last":15.862},{"interface":"front","average":{"1min":17.468,"5min":16.356999999999999,"15min":22.004000000000001},"min":{"1min":13.864000000000001,"5min":10.156000000000001,"15min":8.4390000000000001},"max":{"1min":37.262999999999998,"5min":37.262999999999998,"15min":253.56100000000001},"last":17.404}]},{"osd":5,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":17.428999999999998,"5min":15.695,"15min":22.879999999999999},"min":{"1min":10.804,"5min":9.5199999999999996,"15min":6.6349999999999998},"max":{"1min":35.213999999999999,"5min":35.213999999999999,"15min":336.55599999999998},"last":16.579000000000001},{"interface":"front","average":{"1min":10.366,"5min":11.09,"15min":19.364000000000001},"min":{"1min":7.0019999999999998,"5min":6.4859999999999998,"15min":6.4790000000000001},"max":{"1min":14.853,"5min":23.786000000000001,"15min":357.50900000000001},"last":8.2149999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":14.206,"5min":15.972,"15min":20.510999999999999},"min":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":8.1080000000000005},"max":{"1min":34.234000000000002,"5min":34.234000000000002,"15min":255.52500000000001},"last":9.6910000000000007},{"interface":"front","average":{"1min":19.222000000000001,"5min":15.115,"15min":19.218},"min":{"1min":15.247999999999999,"5min":7.7080000000000002,"15min":7.7080000000000002},"max":{"1min":37.884,"5min":37.884,"15min":312.791},"last":18.312000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":15.161,"5min":15.696,"15min":23.895},"min":{"1min":9.7590000000000003,"5min":9.6940000000000008,"15min":8.7330000000000005},"max":{"1min":23.195,"5min":23.195,"15min":413.48899999999998},"last":13.362},{"interface":"front","average":{"1min":17.57,"5min":16.561,"15min":21.308},"min":{"1min":9.4770000000000003,"5min":9.0809999999999995,"15min":6.6820000000000004},"max":{"1min":30.414999999999999,"5min":30.414999999999999,"15min":417.952},"last":15.034000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575241,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70972,"kb_used_data":7432,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94300868,"statfs":{"total":96636764160,"available":96564088832,"internally_reserved":0,"allocated":7610368,"data_stored":4206728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":13.364000000000001,"5min":11.880000000000001,"15min":16.117000000000001},"min":{"1min":8.8580000000000005,"5min":5.2729999999999997,"15min":4.3360000000000003},"max":{"1min":39.423000000000002,"5min":39.423000000000002,"15min":203.19300000000001},"last":8.9830000000000005},{"interface":"front","average":{"1min":17.373999999999999,"5min":15.071999999999999,"15min":18.241},"min":{"1min":11.513,"5min":6.4269999999999996,"15min":4.4199999999999999},"max":{"1min":40.698999999999998,"5min":40.698999999999998,"15min":204.13},"last":14.613}]},{"osd":1,"last update":"Fri Dec 1 17:35:57 2023","interfaces":[{"interface":"back","average":{"1min":7.9080000000000004,"5min":8.2289999999999992,"15min":15.226000000000001},"min":{"1min":6.1139999999999999,"5min":5.2300000000000004,"15min":5.0369999999999999},"max":{"1min":10.85,"5min":17.564,"15min":167.82900000000001},"last":6.9969999999999999},{"interface":"front","average":{"1min":15.026,"5min":14.375999999999999,"15min":16.369},"min":{"1min":11.634,"5min":7.4870000000000001,"15min":5.484},"max":{"1min":22.280999999999999,"5min":22.280999999999999,"15min":154.137},"last":12.805}]},{"osd":2,"last update":"Fri Dec 1 17:36:23 2023","interfaces":[{"interface":"back","average":{"1min":11.753,"5min":10.013999999999999,"15min":16.608000000000001},"min":{"1min":5.8700000000000001,"5min":5.1820000000000004,"15min":5.1820000000000004},"max":{"1min":26.274999999999999,"5min":26.274999999999999,"15min":230.40600000000001},"last":13.621},{"interface":"front","average":{"1min":14.275,"5min":12.394,"15min":17.012},"min":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"max":{"1min":29.861000000000001,"5min":29.861000000000001,"15min":268.68000000000001},"last":9.5510000000000002}]},{"osd":3,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":13.083,"15min":20.297000000000001},"min":{"1min":8.1359999999999992,"5min":6.5110000000000001,"15min":6.5110000000000001},"max":{"1min":85.724000000000004,"5min":85.724000000000004,"15min":290.15300000000002},"last":8.1999999999999993},{"interface":"front","average":{"1min":14.601000000000001,"5min":11.683,"15min":20.149000000000001},"min":{"1min":6.6319999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"max":{"1min":83.183000000000007,"5min":83.183000000000007,"15min":305.41399999999999},"last":6.1669999999999998}]},{"osd":4,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":14.577,"5min":14.051,"15min":18.489000000000001},"min":{"1min":10.007999999999999,"5min":8.5489999999999995,"15min":6.9299999999999997},"max":{"1min":17.609999999999999,"5min":22.071000000000002,"15min":149.93000000000001},"last":11.746},{"interface":"front","average":{"1min":17.727,"5min":17.503,"15min":23.488},"min":{"1min":14.632,"5min":11.496,"15min":9.7810000000000006},"max":{"1min":26.896999999999998,"5min":27.422000000000001,"15min":239.37},"last":16.341000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":17.140000000000001,"5min":13.266,"15min":20.885000000000002},"min":{"1min":10.646000000000001,"5min":7.3209999999999997,"15min":6.8460000000000001},"max":{"1min":26.053999999999998,"5min":26.053999999999998,"15min":282.31200000000001},"last":17.155000000000001},{"interface":"front","average":{"1min":9.9749999999999996,"5min":13.909000000000001,"15min":19.908999999999999},"min":{"1min":6.8929999999999998,"5min":6.1529999999999996,"15min":6.1529999999999996},"max":{"1min":17.657,"5min":20.282,"15min":193.83600000000001},"last":11.114000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":18.216999999999999,"5min":17.367999999999999,"15min":20.030999999999999},"min":{"1min":13.897,"5min":11.176,"15min":6.625},"max":{"1min":24.832999999999998,"5min":24.832999999999998,"15min":209.25},"last":15.367000000000001},{"interface":"front","average":{"1min":19.565999999999999,"5min":19.302,"15min":24.709},"min":{"1min":15.249000000000001,"5min":14.266999999999999,"15min":8.9420000000000002},"max":{"1min":27.905000000000001,"5min":27.905000000000001,"15min":193.184},"last":17.757000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:36:29.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.679+0000 7f507bda9640 10 monclient: get_monmap_and_config 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.679+0000 7f507bda9640 10 monclient: build_initial_monmap 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.679+0000 7f507bda9640 10 monclient: monmap: 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:29.680489+0000 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:29.680489+0000 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:29.679 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.680 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:29.680 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:29.680 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:29.680 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:29.680 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:29.680 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.680+0000 7f507bda9640 1 Processor -- start 2023-12-01T17:36:29.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.680+0000 7f507bda9640 1 -- start start 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.680+0000 7f507bda9640 10 monclient: init 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50740a8a20 0x7f507413eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient(hunting): picked mon.noname-a con 0x7f50740a8a20 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f50740a8e00 0x7f507413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient(hunting): picked mon.noname-c con 0x7f50740a8e00 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50741423b0 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50740b21f0 con 0x7f50741423b0 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50740a9cc0 con 0x7f50740a8a20 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5074142f50 con 0x7f50740a8e00 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f507bda9640 10 monclient(hunting): authenticate will time out at 3320.918600s 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f5079b45640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f50740a8a20 0x7f507413eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.681+0000 7f5079344640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f50740a8e00 0x7f507413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f50740a8e00 0x7f507413f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47694/0 (socket says 172.21.15.67:47694) 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 1 -- 172.21.15.67:0/3342961037 learned_addr learned my addr 172.21.15.67:0/3342961037 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 10 monclient(hunting): get_auth_request con 0x7f50740a8e00 auth_method 0 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079b45640 10 monclient(hunting): get_auth_request con 0x7f50740a8a20 auth_method 0 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079b45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079b45640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079b45640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.682+0000 7f5079344640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.683+0000 7f5079b45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.683+0000 7f5079b45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.683+0000 7f5079b45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 10 monclient(hunting): handle_auth_done global_id 23149 payload 274 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 1 -- 172.21.15.67:0/3342961037 >> v2:172.21.15.67:3300/0 conn(0x7f50740a8a20 msgr2=0x7f507413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 1 --2- 172.21.15.67:0/3342961037 >> v2:172.21.15.67:3300/0 conn(0x7f50740a8a20 0x7f507413eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 1 -- 172.21.15.67:0/3342961037 >> v1:172.21.15.67:6789/0 conn(0x7f50741423b0 legacy=0x7f5074142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 10 monclient: _finish_hunting 0 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 1 monclient: found mon.noname-c 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.684+0000 7f5079344640 1 -- 172.21.15.67:0/3342961037 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507413f930 con 0x7f50740a8e00 2023-12-01T17:36:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.686+0000 7f5079344640 1 --2- 172.21.15.67:0/3342961037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f507413f3f0 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 crypto rx=0x7f50740b5370 tx=0x7f506402b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89089b9a841f6244 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 1 -- 172.21.15.67:0/3342961037 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50640317e0 con 0x7f50740a8e00 2023-12-01T17:36:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: dump: 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: _finish_auth 0 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: _check_auth_tickets 2023-12-01T17:36:29.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 1 -- 172.21.15.67:0/3342961037 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5064031940 con 0x7f50740a8e00 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 1 -- 172.21.15.67:0/3342961037 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064031cd0 con 0x7f50740a8e00 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 5 monclient: authenticate success, global_id 23149 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 4 set_mon_vals no callback set 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f5078b43640 10 monclient: dump: 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.691 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.692 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.692 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.692 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.692 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.692 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 10 monclient: shutdown 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 1 -- 172.21.15.67:0/3342961037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 msgr2=0x7f507413f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 1 --2- 172.21.15.67:0/3342961037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f507413f3f0 secure :-1 s=READY pgs=5882 cs=0 l=1 rev1=1 crypto rx=0x7f50740b5370 tx=0x7f506402b880 comp rx=0 tx=0).stop 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 1 -- 172.21.15.67:0/3342961037 shutdown_connections 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 1 --2- 172.21.15.67:0/3342961037 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f507413f3f0 unknown :-1 s=CLOSED pgs=5882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 1 --2- 172.21.15.67:0/3342961037 >> v2:172.21.15.67:3300/0 conn(0x7f50740a8a20 0x7f507413eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.690+0000 7f507bda9640 1 -- 172.21.15.67:0/3342961037 >> 172.21.15.67:0/3342961037 conn(0x7f50740b7870 msgr2=0x7f50740b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.691+0000 7f507bda9640 1 -- 172.21.15.67:0/3342961037 shutdown_connections 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.691+0000 7f507bda9640 1 -- 172.21.15.67:0/3342961037 wait complete. 2023-12-01T17:36:29.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.691+0000 7f507bda9640 10 monclient: build_initial_monmap 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.691+0000 7f507bda9640 10 monclient: monmap: 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.691+0000 7f507bda9640 1 librados: starting msgr at 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.691+0000 7f507bda9640 1 librados: starting objecter 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 Processor -- start 2023-12-01T17:36:29.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 -- start start 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 librados: setting wanted keys 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 librados: calling monclient init 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient: init 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50740a8a20 0x7f5074133150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): picked mon.noname-a con 0x7f50740a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f5074134c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): picked mon.noname-b con 0x7f50740a8e00 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50741423b0 0x7f50741d73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): picked mon.noname-c con 0x7f50741423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50741345b0 con 0x7f50740a8a20 2023-12-01T17:36:29.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f507413c180 con 0x7f50740a8e00 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50741347b0 con 0x7f50741423b0 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.692+0000 7f507bda9640 10 monclient(hunting): authenticate will time out at 3320.930009s 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079b45640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50740a8a20 0x7f5074133150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f5074134c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f507a346640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50741423b0 0x7f50741d73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f5074134c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47710/0 (socket says 172.21.15.67:47710) 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 1 -- 172.21.15.67:0/2536202734 learned_addr learned my addr 172.21.15.67:0/2536202734 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 10 monclient(hunting): get_auth_request con 0x7f50740a8e00 auth_method 0 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079b45640 10 monclient(hunting): get_auth_request con 0x7f50740a8a20 auth_method 0 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079b45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079b45640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079b45640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f507a346640 10 monclient(hunting): get_auth_request con 0x7f50741423b0 auth_method 0 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f507a346640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f507a346640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f507a346640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.693+0000 7f5079344640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.694+0000 7f5079b45640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.694+0000 7f5079b45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.694+0000 7f5079b45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.694+0000 7f507a346640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:29.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.694+0000 7f507a346640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.694+0000 7f507a346640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 10 monclient(hunting): handle_auth_done global_id 23152 payload 824 2023-12-01T17:36:29.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 1 -- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50741423b0 msgr2=0x7f50741d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50741423b0 0x7f50741d73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 1 -- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50740a8a20 msgr2=0x7f5074133150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50740a8a20 0x7f5074133150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 10 monclient: _finish_hunting 0 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 1 monclient: found mon.noname-b 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.696+0000 7f5079344640 1 -- 172.21.15.67:0/2536202734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50640094d0 con 0x7f50740a8e00 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.698+0000 7f5079344640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f5074134c40 secure :-1 s=READY pgs=5883 cs=0 l=1 rev1=1 crypto rx=0x7f506402b8c0 tx=0x7f50640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.698+0000 7f506a7fc640 10 client.?.objecter ms_handle_connect 0x7f50740a8e00 2023-12-01T17:36:29.701 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.698+0000 7f506a7fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:29.702 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 1 -- 172.21.15.67:0/2536202734 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064032880 con 0x7f50740a8e00 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: dump: 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.703 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: _finish_auth 0 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 1 -- 172.21.15.67:0/2536202734 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50640329e0 con 0x7f50740a8e00 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 1 -- 172.21.15.67:0/2536202734 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5064032d90 con 0x7f50740a8e00 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 5 monclient: authenticate success, global_id 23152 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 10 monclient: _renew_subs 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50741362e0 con 0x7f50740a8e00 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506bfff640 4 set_mon_vals no callback set 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f506a7fc640 10 monclient: dump: 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:29.704 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 10 client.23152.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 10 monclient: _renew_subs 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f507413df80 con 0x7f50740a8e00 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.702+0000 7f507bda9640 1 librados: init done 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.703+0000 7f507bda9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:29.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.703+0000 7f507bda9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.703+0000 7f507bda9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:29.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.703+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f507404e840 con 0x7f50740a8e00 2023-12-01T17:36:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.944+0000 7f506a7fc640 1 -- 172.21.15.67:0/2536202734 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f506404e450 con 0x7f50740a8e00 2023-12-01T17:36:29.943 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.944+0000 7f506a7fc640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f505806c870 0x7f505806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.944+0000 7f5079b45640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f505806c870 0x7f505806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.944+0000 7f5079b45640 10 monclient: get_auth_request con 0x7f505806c870 auth_method 0 2023-12-01T17:36:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.944+0000 7f5079b45640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f505806c870 0x7f505806ed30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5070009b40 tx=0x7f5070009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:29.944 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.944+0000 7f506a7fc640 10 client.23152.objecter ms_handle_connect 0x7f505806c870 2023-12-01T17:36:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.946+0000 7f506a7fc640 1 -- 172.21.15.67:0/2536202734 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1006..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f506407aa40 con 0x7f50740a8e00 2023-12-01T17:36:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.946+0000 7f506a7fc640 10 client.23152.objecter ms_dispatch 0x7f5074000b90 osd_map(1099..1099 src has 1006..1099) v4 2023-12-01T17:36:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.946+0000 7f506a7fc640 3 client.23152.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.946+0000 7f506a7fc640 3 client.23152.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:29.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.946+0000 7f506a7fc640 20 client.23152.objecter dump_active .. 0 homeless 2023-12-01T17:36:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.956+0000 7f506a7fc640 1 -- 172.21.15.67:0/2536202734 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5064076d80 con 0x7f50740a8e00 2023-12-01T17:36:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.956+0000 7f506a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:29.956 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:29.956+0000 7f506a7fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:30.092 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:43.874 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.097 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:43.878 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.112 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:43.894 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.145 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:43.906 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.148 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:43.909 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.162 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:43.923 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.172 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:43.953 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.175 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:43.957 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.198 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:43.976 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.232+0000 7f507bda9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:36:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.232+0000 7f507bda9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:36:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.232+0000 7f507bda9640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:30.232 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.232+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5074136800 con 0x7f50740a8e00 2023-12-01T17:36:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.339+0000 7f506a7fc640 1 -- 172.21.15.67:0/2536202734 <== mon.1 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1099) v1 ==== 74+0+17629 (secure 0 0 0) 0x7f506407a3a0 con 0x7f50740a8e00 2023-12-01T17:36:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.339+0000 7f506a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-12-01T17:36:30.339 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.339+0000 7f506a7fc640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:36:30.340 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:36:30.340 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":1099,"fsid":"2c621237-d4b8-4504-931d-6a9429822e96","created":"2023-12-01T17:03:04.093142+0000","modified":"2023-12-01T17:36:26.989672+0000","last_up_change":"2023-12-01T17:03:52.677540+0000","last_in_change":"2023-12-01T17:03:09.418337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":113,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-01T17:03:50.595091+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-01T17:03:58.823298+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"ceph_test_rados_api_asio","create_time":"2023-12-01T17:05:03.135640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"fbc446fc-b047-4553-83da-1ab39870ff2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6811","nonce":358761697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6813","nonce":358761697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6817","nonce":358761697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":358761697},{"type":"v1","addr":"172.21.15.67:6815","nonce":358761697}]},"public_addr":"172.21.15.67:6811/358761697","cluster_addr":"172.21.15.67:6813/358761697","heartbeat_back_addr":"172.21.15.67:6817/358761697","heartbeat_front_addr":"172.21.15.67:6815/358761697","state":["exists","up"]},{"osd":1,"uuid":"333c878f-1386-495c-a6d6-1fd5e68c005e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":17,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6827","nonce":866081726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6829","nonce":866081726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6833","nonce":866081726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":866081726},{"type":"v1","addr":"172.21.15.67:6831","nonce":866081726}]},"public_addr":"172.21.15.67:6827/866081726","cluster_addr":"172.21.15.67:6829/866081726","heartbeat_back_addr":"172.21.15.67:6833/866081726","heartbeat_front_addr":"172.21.15.67:6831/866081726","state":["exists","up"]},{"osd":2,"uuid":"f72d775b-8e17-4768-87e3-6cc7a2bdb6a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6819","nonce":2342127620}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6821","nonce":2342127620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6825","nonce":2342127620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":2342127620},{"type":"v1","addr":"172.21.15.67:6823","nonce":2342127620}]},"public_addr":"172.21.15.67:6819/2342127620","cluster_addr":"172.21.15.67:6821/2342127620","heartbeat_back_addr":"172.21.15.67:6825/2342127620","heartbeat_front_addr":"172.21.15.67:6823/2342127620","state":["exists","up"]},{"osd":3,"uuid":"ad66de04-1a5c-47a6-9e6e-68b141fac874","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6803","nonce":1625009733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6805","nonce":1625009733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6809","nonce":1625009733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1625009733},{"type":"v1","addr":"172.21.15.67:6807","nonce":1625009733}]},"public_addr":"172.21.15.67:6803/1625009733","cluster_addr":"172.21.15.67:6805/1625009733","heartbeat_back_addr":"172.21.15.67:6809/1625009733","heartbeat_front_addr":"172.21.15.67:6807/1625009733","state":["exists","up"]},{"osd":4,"uuid":"dfce6366-008d-476d-b47a-f2d56833077c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6817","nonce":2070368587}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6819","nonce":2070368587}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6823","nonce":2070368587}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2070368587},{"type":"v1","addr":"172.21.15.92:6821","nonce":2070368587}]},"public_addr":"172.21.15.92:6817/2070368587","cluster_addr":"172.21.15.92:6819/2070368587","heartbeat_back_addr":"172.21.15.92:6823/2070368587","heartbeat_front_addr":"172.21.15.92:6821/2070368587","state":["exists","up"]},{"osd":5,"uuid":"4fb3a3b6-1879-4cc9-9a36-07d8468de812","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6801","nonce":1636405563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6803","nonce":1636405563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6807","nonce":1636405563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":1636405563},{"type":"v1","addr":"172.21.15.92:6805","nonce":1636405563}]},"public_addr":"172.21.15.92:6801/1636405563","cluster_addr":"172.21.15.92:6803/1636405563","heartbeat_back_addr":"172.21.15.92:6807/1636405563","heartbeat_front_addr":"172.21.15.92:6805/1636405563","state":["exists","up"]},{"osd":6,"uuid":"76c95f42-d993-4ac3-85b3-f6357c571f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6825","nonce":162801920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6827","nonce":162801920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6831","nonce":162801920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":162801920},{"type":"v1","addr":"172.21.15.92:6829","nonce":162801920}]},"public_addr":"172.21.15.92:6825/162801920","cluster_addr":"172.21.15.92:6827/162801920","heartbeat_back_addr":"172.21.15.92:6831/162801920","heartbeat_front_addr":"172.21.15.92:6829/162801920","state":["exists","up"]},{"osd":7,"uuid":"465c0392-a311-4499-b75a-7b5ad156d99e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1096,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6809","nonce":2695530702}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6811","nonce":2695530702}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6815","nonce":2695530702}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":2695530702},{"type":"v1","addr":"172.21.15.92:6813","nonce":2695530702}]},"public_addr":"172.21.15.92:6809/2695530702","cluster_addr":"172.21.15.92:6811/2695530702","heartbeat_back_addr":"172.21.15.92:6815/2695530702","heartbeat_front_addr":"172.21.15.92:6813/2695530702","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.603058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.949751+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:49.632542+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:48.910362+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.308141+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.751004+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:47.105081+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-01T17:03:46.199584+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":8,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":16,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":23,"snaps":[{"begin":1,"length":1}]},{"pool":55,"snaps":[{"begin":1,"length":1}]},{"pool":74,"snaps":[{"begin":11,"length":1}]},{"pool":85,"snaps":[{"begin":1,"length":1}]},{"pool":92,"snaps":[{"begin":1,"length":2}]},{"pool":113,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":116,"snaps":[{"begin":3,"length":1}]},{"pool":124,"snaps":[{"begin":2,"length":2}]},{"pool":129,"snaps":[{"begin":6,"length":2}]},{"pool":137,"snaps":[{"begin":2,"length":3}]},{"pool":147,"snaps":[{"begin":2,"length":2}]},{"pool":150,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":163,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":246,"snaps":[{"begin":12,"length":1}]},{"pool":257,"snaps":[{"begin":14,"length":1}]},{"pool":259,"snaps":[{"begin":17,"length":1}]},{"pool":260,"snaps":[{"begin":19,"length":2}]},{"pool":263,"snaps":[{"begin":24,"length":1}]},{"pool":308,"snaps":[{"begin":2,"length":1}]},{"pool":309,"snaps":[{"begin":4,"length":1}]},{"pool":312,"snaps":[{"begin":6,"length":1}]},{"pool":316,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-01T17:36:30.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 10 librados: watch_flush enter 2023-12-01T17:36:30.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 10 librados: watch_flush exit 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 20 client.23152.objecter shutdown clearing up homeless session... 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 10 client.23152.objecter successfully canceled tick 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f505806c870 msgr2=0x7f505806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f505806c870 0x7f505806ed30 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f5070009b40 tx=0x7f5070009190 comp rx=0 tx=0).stop 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 10 monclient: shutdown 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 msgr2=0x7f5074134c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f5074134c40 secure :-1 s=READY pgs=5883 cs=0 l=1 rev1=1 crypto rx=0x7f506402b8c0 tx=0x7f50640046c0 comp rx=0 tx=0).stop 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 shutdown_connections 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.341+0000 7f507bda9640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f505806c870 0x7f505806ed30 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.342+0000 7f507bda9640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f50741423b0 0x7f50741d73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.342+0000 7f507bda9640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f50740a8e00 0x7f5074134c40 unknown :-1 s=CLOSED pgs=5883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.342+0000 7f507bda9640 1 --2- 172.21.15.67:0/2536202734 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50740a8a20 0x7f5074133150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:30.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.342+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 >> 172.21.15.67:0/2536202734 conn(0x7f50740b7870 msgr2=0x7f50741380c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:30.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.342+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 shutdown_connections 2023-12-01T17:36:30.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.342+0000 7f507bda9640 1 -- 172.21.15.67:0/2536202734 wait complete. 2023-12-01T17:36:30.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:30.342+0000 7f507bda9640 1 librados: shutdown 2023-12-01T17:36:31.353 INFO:tasks.ceph:Scrubbing osd.0 2023-12-01T17:36:31.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:31.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.465+0000 7fef4256c640 10 monclient: get_monmap_and_config 2023-12-01T17:36:31.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.465+0000 7fef4256c640 10 monclient: build_initial_monmap 2023-12-01T17:36:31.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.465+0000 7fef4256c640 10 monclient: monmap: 2023-12-01T17:36:31.465 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:31.465 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.465 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:31.466681+0000 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:31.466681+0000 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:31.466 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.466+0000 7fef4256c640 1 Processor -- start 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.466+0000 7fef4256c640 1 -- start start 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.466+0000 7fef4256c640 10 monclient: init 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef3c0a8a20 0x7fef3c13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient(hunting): picked mon.noname-c con 0x7fef3c0a8a20 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:31.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef3c0a8e00 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c1423b0 0x7fef3c142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient(hunting): picked mon.noname-b con 0x7fef3c1423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef3c0b21f0 con 0x7fef3c0a8e00 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef3c0a9cc0 con 0x7fef3c0a8a20 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c0a9ed0 con 0x7fef3c1423b0 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef4256c640 10 monclient(hunting): authenticate will time out at 3322.704736s 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.467+0000 7fef3bfff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fef3c0a8a20 0x7fef3c13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3b7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fef3c0a8e00 0x7fef3c13f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:49344/0 (socket says 172.21.15.67:49344) 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3b7fe640 1 -- 172.21.15.67:0/142352294 learned_addr learned my addr 172.21.15.67:0/142352294 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef40b09640 1 --2- 172.21.15.67:0/142352294 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c1423b0 0x7fef3c142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3bfff640 10 monclient(hunting): get_auth_request con 0x7fef3c0a8a20 auth_method 0 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef40b09640 10 monclient(hunting): get_auth_request con 0x7fef3c1423b0 auth_method 0 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef40b09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef40b09640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef40b09640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.468+0000 7fef3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.469+0000 7fef40b09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.469+0000 7fef40b09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.469+0000 7fef40b09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 10 monclient(hunting): handle_auth_done global_id 23155 payload 274 2023-12-01T17:36:31.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 1 -- 172.21.15.67:0/142352294 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c1423b0 msgr2=0x7fef3c142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 1 --2- 172.21.15.67:0/142352294 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c1423b0 0x7fef3c142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 1 -- 172.21.15.67:0/142352294 >> v1:172.21.15.67:6789/0 conn(0x7fef3c0a8e00 legacy=0x7fef3c13f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 10 monclient: _finish_hunting 0 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 1 monclient: found mon.noname-c 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.470+0000 7fef3bfff640 1 -- 172.21.15.67:0/142352294 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3c0aac70 con 0x7fef3c0a8a20 2023-12-01T17:36:31.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.472+0000 7fef3bfff640 1 --2- 172.21.15.67:0/142352294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c0a8a20 0x7fef3c13eeb0 secure :-1 s=READY pgs=5884 cs=0 l=1 rev1=1 crypto rx=0x7fef300097a0 tx=0x7fef3002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6f50d9488cc5768 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:31.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.475+0000 7fef3affd640 1 -- 172.21.15.67:0/142352294 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3002f050 con 0x7fef3c0a8a20 2023-12-01T17:36:31.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:31.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: dump: 2023-12-01T17:36:31.475 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: _finish_auth 0 2023-12-01T17:36:31.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 1 -- 172.21.15.67:0/142352294 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef30031700 con 0x7fef3c0a8a20 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 5 monclient: authenticate success, global_id 23155 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 4 set_mon_vals no callback set 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 1 -- 172.21.15.67:0/142352294 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30031a90 con 0x7fef3c0a8a20 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef3affd640 10 monclient: dump: 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.478 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.478 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.478 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.478 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.478 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.478 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 10 monclient: shutdown 2023-12-01T17:36:31.481 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 -- 172.21.15.67:0/142352294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c0a8a20 msgr2=0x7fef3c13eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 --2- 172.21.15.67:0/142352294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c0a8a20 0x7fef3c13eeb0 secure :-1 s=READY pgs=5884 cs=0 l=1 rev1=1 crypto rx=0x7fef300097a0 tx=0x7fef3002b7a0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 -- 172.21.15.67:0/142352294 shutdown_connections 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 --2- 172.21.15.67:0/142352294 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c1423b0 0x7fef3c142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 --2- 172.21.15.67:0/142352294 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c0a8a20 0x7fef3c13eeb0 unknown :-1 s=CLOSED pgs=5884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 -- 172.21.15.67:0/142352294 >> 172.21.15.67:0/142352294 conn(0x7fef3c0b7870 msgr2=0x7fef3c0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 -- 172.21.15.67:0/142352294 shutdown_connections 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.476+0000 7fef4256c640 1 -- 172.21.15.67:0/142352294 wait complete. 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.477+0000 7fef4256c640 10 monclient: build_initial_monmap 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.477+0000 7fef4256c640 10 monclient: monmap: 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:31.482 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.477+0000 7fef4256c640 1 librados: starting msgr at 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.477+0000 7fef4256c640 1 librados: starting objecter 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.477+0000 7fef4256c640 1 Processor -- start 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 -- start start 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 librados: setting wanted keys 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 librados: calling monclient init 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient: init 2023-12-01T17:36:31.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c0a8a20 0x7fef3c138560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): picked mon.noname-c con 0x7fef3c0a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef3c0a8e00 0x7fef3c145600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): picked mon.noname-a con 0x7fef3c0a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c1423b0 0x7fef3c145b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): picked mon.noname-b con 0x7fef3c1423b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c143800 con 0x7fef3c0a8e00 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c146080 con 0x7fef3c1423b0 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c144c20 con 0x7fef3c0a8a20 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:31.484 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.478+0000 7fef4256c640 10 monclient(hunting): authenticate will time out at 3322.716205s 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.479+0000 7fef3b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef3c0a8e00 0x7fef3c145600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.479+0000 7fef3bfff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c0a8a20 0x7fef3c138560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3b7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef3c0a8e00 0x7fef3c145600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56640/0 (socket says 172.21.15.67:56640) 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3b7fe640 1 -- 172.21.15.67:0/3091174212 learned_addr learned my addr 172.21.15.67:0/3091174212 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3b7fe640 10 monclient(hunting): get_auth_request con 0x7fef3c0a8e00 auth_method 0 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3b7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3b7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3bfff640 10 monclient(hunting): get_auth_request con 0x7fef3c0a8a20 auth_method 0 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3bfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3bfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.480+0000 7fef3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.481+0000 7fef40b09640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c1423b0 0x7fef3c145b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.481+0000 7fef40b09640 10 monclient(hunting): get_auth_request con 0x7fef3c1423b0 auth_method 0 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.481+0000 7fef40b09640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.481+0000 7fef40b09640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.481+0000 7fef40b09640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.482+0000 7fef3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.482+0000 7fef3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.482+0000 7fef3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.482+0000 7fef40b09640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.482+0000 7fef40b09640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.486 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.482+0000 7fef40b09640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 10 monclient(hunting): handle_auth_done global_id 17915 payload 824 2023-12-01T17:36:31.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 1 -- 172.21.15.67:0/3091174212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c1423b0 msgr2=0x7fef3c145b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fef3c1423b0 0x7fef3c145b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.487 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 1 -- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef3c0a8e00 msgr2=0x7fef3c145600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fef3c0a8e00 0x7fef3c145600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 10 monclient: _finish_hunting 0 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 1 monclient: found mon.noname-c 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.67:3301/0 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.484+0000 7fef3bfff640 1 -- 172.21.15.67:0/3091174212 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef30009400 con 0x7fef3c0a8a20 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.486+0000 7fef3bfff640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c0a8a20 0x7fef3c138560 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7fef30033000 tx=0x7fef300046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.486+0000 7fef38ff9640 10 client.?.objecter ms_handle_connect 0x7fef3c0a8a20 2023-12-01T17:36:31.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.486+0000 7fef38ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30031500 con 0x7fef3c0a8a20 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: dump: 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.491 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: _finish_auth 0 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef30033a30 con 0x7fef3c0a8a20 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef4256c640 5 monclient: authenticate success, global_id 17915 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef4256c640 10 monclient: _renew_subs 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef4256c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3c144f00 con 0x7fef3c0a8a20 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:31.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30033de0 con 0x7fef3c0a8a20 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef38ff9640 10 monclient: dump: 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.493 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.490+0000 7fef3a7fc640 4 set_mon_vals no callback set 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 10 client.17915.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 10 monclient: _renew_subs 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3c143c90 con 0x7fef3c0a8a20 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 1 librados: init done 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 10 client.17915.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 15 client.17915.objecter _session_command_op_assign -1 1 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 20 client.17915.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 10 client.17915.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.67:3301/0 2023-12-01T17:36:31.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.491+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fef3c13fb50 con 0x7fef3c0a8a20 2023-12-01T17:36:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.514+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 <== mon.2 v2:172.21.15.67:3301/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fef3003a040 con 0x7fef3c0a8a20 2023-12-01T17:36:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.514+0000 7fef38ff9640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fef2806c820 0x7fef2806ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.515+0000 7fef3b7fe640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fef2806c820 0x7fef2806ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.515+0000 7fef3b7fe640 10 monclient: get_auth_request con 0x7fef2806c820 auth_method 0 2023-12-01T17:36:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.515+0000 7fef3b7fe640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fef2806c820 0x7fef2806ece0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0b15d0 tx=0x7fef2c007ee0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:31.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.515+0000 7fef38ff9640 10 client.17915.objecter ms_handle_connect 0x7fef2806c820 2023-12-01T17:36:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.519+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 <== mon.2 v2:172.21.15.67:3301/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fef3007a0c0 con 0x7fef3c0a8a20 2023-12-01T17:36:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.519+0000 7fef38ff9640 10 client.17915.objecter ms_dispatch 0x7fef3c000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.519+0000 7fef38ff9640 3 client.17915.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.519+0000 7fef38ff9640 3 client.17915.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:31.519 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 10 client.17915.objecter checking command 1 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fef28074410 0x7fef280768b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 20 client.17915.objecter _get_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 20 client.17915.objecter put_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 15 client.17915.objecter _session_command_op_remove -1 1 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 20 client.17915.objecter _get_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 20 client.17915.objecter get_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 15 client.17915.objecter _session_command_op_assign 0 1 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 20 client.17915.objecter put_session s=0x7fef28073d40 osd=0 4 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 10 client.17915.objecter _send_command 1 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 --> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef28076f80 con 0x7fef28074410 2023-12-01T17:36:31.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef38ff9640 20 client.17915.objecter dump_active .. 0 homeless 2023-12-01T17:36:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.520+0000 7fef40b09640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fef28074410 0x7fef280768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.521+0000 7fef40b09640 10 monclient: get_auth_request con 0x7fef28074410 auth_method 0 2023-12-01T17:36:31.523 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.523+0000 7fef38ff9640 10 client.17915.objecter ms_handle_connect 0x7fef28074410 2023-12-01T17:36:31.523 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.523+0000 7fef40b09640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fef28074410 0x7fef280768b0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:31.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.549+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 <== osd.0 v2:172.21.15.67:6810/358761697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fef28076f80 con 0x7fef28074410 2023-12-01T17:36:31.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.549+0000 7fef38ff9640 10 client.17915.objecter ms_dispatch 0x7fef3c000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:31.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.549+0000 7fef38ff9640 10 client.17915.objecter _finish_command 1 = system:0 2023-12-01T17:36:31.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.549+0000 7fef38ff9640 20 client.17915.objecter put_session s=0x7fef28073d40 osd=0 4 2023-12-01T17:36:31.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.549+0000 7fef38ff9640 15 client.17915.objecter _session_command_op_remove 0 1 2023-12-01T17:36:31.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 10 client.17915.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 15 client.17915.objecter _session_command_op_assign -1 2 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 20 client.17915.objecter _get_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 20 client.17915.objecter put_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 20 client.17915.objecter _get_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 15 client.17915.objecter _session_command_op_remove -1 2 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 20 client.17915.objecter get_session s=0x7fef28073d40 osd=0 3 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 15 client.17915.objecter _session_command_op_assign 0 2 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 20 client.17915.objecter put_session s=0x7fef28073d40 osd=0 4 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 10 client.17915.objecter _send_command 2 2023-12-01T17:36:31.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.564+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 --> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fef3c04e840 con 0x7fef28074410 2023-12-01T17:36:31.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.577+0000 7fef38ff9640 1 -- 172.21.15.67:0/3091174212 <== osd.0 v2:172.21.15.67:6810/358761697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fef3c04e840 con 0x7fef28074410 2023-12-01T17:36:31.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.577+0000 7fef38ff9640 10 client.17915.objecter ms_dispatch 0x7fef3c000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:31.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.577+0000 7fef38ff9640 10 client.17915.objecter _finish_command 2 = system:0 2023-12-01T17:36:31.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.577+0000 7fef38ff9640 20 client.17915.objecter put_session s=0x7fef28073d40 osd=0 4 2023-12-01T17:36:31.577 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.577+0000 7fef38ff9640 15 client.17915.objecter _session_command_op_remove 0 2 2023-12-01T17:36:31.578 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:31.578 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:31.578 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:31.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 10 librados: watch_flush enter 2023-12-01T17:36:31.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 10 librados: watch_flush exit 2023-12-01T17:36:31.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 10 client.17915.objecter close_session for osd.0 2023-12-01T17:36:31.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fef28074410 msgr2=0x7fef280768b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.579 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6810/358761697,v1:172.21.15.67:6811/358761697] conn(0x7fef28074410 0x7fef280768b0 crc :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 20 client.17915.objecter put_session s=0x7fef28073d40 osd=0 1 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 20 client.17915.objecter shutdown clearing up homeless session... 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 10 client.17915.objecter successfully canceled tick 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fef2806c820 msgr2=0x7fef2806ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fef2806c820 0x7fef2806ece0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fef3c0b15d0 tx=0x7fef2c007ee0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 10 monclient: shutdown 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 20 monclient: shutdown canceling and discarding version request 1 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c0a8a20 msgr2=0x7fef3c138560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 --2- 172.21.15.67:0/3091174212 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fef3c0a8a20 0x7fef3c138560 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7fef30033000 tx=0x7fef300046c0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef40b09640 1 -- 172.21.15.67:0/3091174212 reap_dead start 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 shutdown_connections 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 >> 172.21.15.67:0/3091174212 conn(0x7fef3c0b7870 msgr2=0x7fef3c13d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 shutdown_connections 2023-12-01T17:36:31.580 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 -- 172.21.15.67:0/3091174212 wait complete. 2023-12-01T17:36:31.581 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.578+0000 7fef4256c640 1 librados: shutdown 2023-12-01T17:36:31.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.706+0000 7f2a83689640 10 monclient: get_monmap_and_config 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.706+0000 7f2a83689640 10 monclient: build_initial_monmap 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.706+0000 7f2a83689640 10 monclient: monmap: 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:31.707548+0000 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:31.707548+0000 2023-12-01T17:36:31.706 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:31.707 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.707 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:31.707 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:31.707 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:31.707 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:31.707 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:31.707 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.707+0000 7f2a83689640 1 Processor -- start 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.707+0000 7f2a83689640 1 -- start start 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.707+0000 7f2a83689640 10 monclient: init 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2a7c0a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2a7c0a8e00 0x7f2a7c13f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7c0a8e00 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c1423b0 0x7f2a7c142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient(hunting): picked mon.noname-b con 0x7f2a7c1423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7c0b21f0 con 0x7f2a7c0a8a20 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7c0a9cc0 con 0x7f2a7c0a8e00 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c0a9ed0 con 0x7f2a7c1423b0 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a83689640 10 monclient(hunting): authenticate will time out at 3322.945684s 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.708+0000 7f2a81c26640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c1423b0 0x7f2a7c142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a80c24640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2a7c0a8e00 0x7f2a7c13f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81425640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2a7c0a8a20 0x7f2a7c13eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:36914/0 (socket says 172.21.15.67:36914) 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81425640 1 -- 172.21.15.67:0/3953999237 learned_addr learned my addr 172.21.15.67:0/3953999237 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:31.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81c26640 10 monclient(hunting): get_auth_request con 0x7f2a7c1423b0 auth_method 0 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81c26640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81c26640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a80c24640 10 monclient(hunting): get_auth_request con 0x7f2a7c0a8e00 auth_method 0 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a80c24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a80c24640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a80c24640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.709+0000 7f2a81c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.710+0000 7f2a80c24640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.710+0000 7f2a80c24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.710+0000 7f2a80c24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 10 monclient(hunting): handle_auth_done global_id 17918 payload 274 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 1 -- 172.21.15.67:0/3953999237 >> v2:172.21.15.92:3300/0 conn(0x7f2a7c0a8e00 msgr2=0x7f2a7c13f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 1 --2- 172.21.15.67:0/3953999237 >> v2:172.21.15.92:3300/0 conn(0x7f2a7c0a8e00 0x7f2a7c13f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 1 -- 172.21.15.67:0/3953999237 >> v1:172.21.15.92:6789/0 conn(0x7f2a7c0a8a20 legacy=0x7f2a7c13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 10 monclient: _finish_hunting 0 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 1 monclient: found mon.noname-b 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:31.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.711+0000 7f2a81c26640 1 -- 172.21.15.67:0/3953999237 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c0aac70 con 0x7f2a7c1423b0 2023-12-01T17:36:31.713 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.713+0000 7f2a81c26640 1 --2- 172.21.15.67:0/3953999237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c1423b0 0x7f2a7c142840 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009d90 tx=0x7f2a7402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce242876972bb9b1 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 1 -- 172.21.15.67:0/3953999237 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7402cb30 con 0x7f2a7c1423b0 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: dump: 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.716 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: _finish_auth 0 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 1 -- 172.21.15.67:0/3953999237 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7402cc90 con 0x7f2a7c1423b0 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 1 -- 172.21.15.67:0/3953999237 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74034440 con 0x7f2a7c1423b0 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a83689640 5 monclient: authenticate success, global_id 17918 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a83689640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:31.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a83689640 4 set_mon_vals no callback set 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.716+0000 7f2a73fff640 10 monclient: dump: 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.718 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 10 monclient: shutdown 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 -- 172.21.15.67:0/3953999237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c1423b0 msgr2=0x7f2a7c142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 --2- 172.21.15.67:0/3953999237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c1423b0 0x7f2a7c142840 secure :-1 s=READY pgs=3068 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009d90 tx=0x7f2a7402bd80 comp rx=0 tx=0).stop 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 -- 172.21.15.67:0/3953999237 shutdown_connections 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 --2- 172.21.15.67:0/3953999237 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c1423b0 0x7f2a7c142840 unknown :-1 s=CLOSED pgs=3068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 --2- 172.21.15.67:0/3953999237 >> v2:172.21.15.92:3300/0 conn(0x7f2a7c0a8e00 0x7f2a7c13f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 -- 172.21.15.67:0/3953999237 >> 172.21.15.67:0/3953999237 conn(0x7f2a7c0b7870 msgr2=0x7f2a7c0b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 -- 172.21.15.67:0/3953999237 shutdown_connections 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.717+0000 7f2a83689640 1 -- 172.21.15.67:0/3953999237 wait complete. 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.718+0000 7f2a83689640 10 monclient: build_initial_monmap 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.718+0000 7f2a83689640 10 monclient: monmap: 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:31.722 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 librados: starting msgr at 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 librados: starting objecter 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 Processor -- start 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 -- start start 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 librados: setting wanted keys 2023-12-01T17:36:31.723 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 librados: calling monclient init 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient: init 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c0a8a20 0x7f2a7c138190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): picked mon.noname-c con 0x7f2a7c0a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c0a8e00 0x7f2a7c145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): picked mon.noname-a con 0x7f2a7c0a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a7c1423b0 0x7f2a7c145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): picked mon.noname-b con 0x7f2a7c1423b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c145fa0 con 0x7f2a7c0a8e00 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c1448d0 con 0x7f2a7c1423b0 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:31.724 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c144b40 con 0x7f2a7c0a8a20 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.719+0000 7f2a83689640 10 monclient(hunting): authenticate will time out at 3322.957159s 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a7c1423b0 0x7f2a7c145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81425640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c0a8a20 0x7f2a7c138190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a7c1423b0 0x7f2a7c145a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42604/0 (socket says 172.21.15.67:42604) 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 1 -- 172.21.15.67:0/832059656 learned_addr learned my addr 172.21.15.67:0/832059656 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 10 monclient(hunting): get_auth_request con 0x7f2a7c1423b0 auth_method 0 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a80c24640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c0a8e00 0x7f2a7c145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81425640 10 monclient(hunting): get_auth_request con 0x7f2a7c0a8a20 auth_method 0 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81425640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81425640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.725 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81425640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a80c24640 10 monclient(hunting): get_auth_request con 0x7f2a7c0a8e00 auth_method 0 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a80c24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a80c24640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a80c24640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.720+0000 7f2a81c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.721+0000 7f2a81425640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.721+0000 7f2a81425640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.721+0000 7f2a81425640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.721+0000 7f2a80c24640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.721+0000 7f2a80c24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:31.726 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.721+0000 7f2a80c24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 10 monclient(hunting): handle_auth_done global_id 23167 payload 824 2023-12-01T17:36:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 1 -- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c0a8a20 msgr2=0x7f2a7c138190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c0a8a20 0x7f2a7c138190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 1 -- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c0a8e00 msgr2=0x7f2a7c145520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:31.727 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c0a8e00 0x7f2a7c145520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 10 monclient: _finish_hunting 0 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 1 monclient: found mon.noname-b 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.723+0000 7f2a81c26640 1 -- 172.21.15.67:0/832059656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74009a70 con 0x7f2a7c1423b0 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.725+0000 7f2a81c26640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a7c1423b0 0x7f2a7c145a60 secure :-1 s=READY pgs=5886 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009eb0 tx=0x7f2a74004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.725+0000 7f2a71ffb640 10 client.?.objecter ms_handle_connect 0x7f2a7c1423b0 2023-12-01T17:36:31.728 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.725+0000 7f2a71ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:31.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 1 -- 172.21.15.67:0/832059656 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74009450 con 0x7f2a7c1423b0 2023-12-01T17:36:31.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:31.729 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: dump: 2023-12-01T17:36:31.729 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.729 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.729 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 1 -- 172.21.15.67:0/832059656 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a74040450 con 0x7f2a7c1423b0 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 1 -- 172.21.15.67:0/832059656 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a74040800 con 0x7f2a7c1423b0 2023-12-01T17:36:31.730 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 5 monclient: authenticate success, global_id 23167 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 10 monclient: _renew_subs 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c144e20 con 0x7f2a7c1423b0 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a71ffb640 10 monclient: dump: 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:31.731 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a737fe640 4 set_mon_vals no callback set 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 10 client.23167.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 10 monclient: _renew_subs 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c143390 con 0x7f2a7c1423b0 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.729+0000 7f2a83689640 1 librados: init done 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.730+0000 7f2a83689640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.730+0000 7f2a83689640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.730+0000 7f2a83689640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:31.732 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.730+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c1439b0 con 0x7f2a7c1423b0 2023-12-01T17:36:31.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.969+0000 7f2a71ffb640 1 -- 172.21.15.67:0/832059656 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f2a74040960 con 0x7f2a7c1423b0 2023-12-01T17:36:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.970+0000 7f2a71ffb640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2a6006c820 0x7f2a6006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.970+0000 7f2a81425640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2a6006c820 0x7f2a6006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.970+0000 7f2a81425640 10 monclient: get_auth_request con 0x7f2a6006c820 auth_method 0 2023-12-01T17:36:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.970+0000 7f2a71ffb640 10 client.23167.objecter ms_handle_connect 0x7f2a6006c820 2023-12-01T17:36:31.970 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.970+0000 7f2a81425640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2a6006c820 0x7f2a6006ece0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2a78007fd0 tx=0x7f2a78007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:31.971 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.972+0000 7f2a71ffb640 1 -- 172.21.15.67:0/832059656 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f2a7407b220 con 0x7f2a7c1423b0 2023-12-01T17:36:31.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.972+0000 7f2a71ffb640 10 client.23167.objecter ms_dispatch 0x7f2a7c000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:31.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.972+0000 7f2a71ffb640 3 client.23167.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:31.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.972+0000 7f2a71ffb640 3 client.23167.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:31.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.972+0000 7f2a71ffb640 20 client.23167.objecter dump_active .. 0 homeless 2023-12-01T17:36:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.982+0000 7f2a71ffb640 1 -- 172.21.15.67:0/832059656 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a74077560 con 0x7f2a7c1423b0 2023-12-01T17:36:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.982+0000 7f2a71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:31.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:31.982+0000 7f2a71ffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.161+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a7c1439b0 con 0x7f2a6006c820 2023-12-01T17:36:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.161+0000 7f2a71ffb640 1 -- 172.21.15.67:0/832059656 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2a7c1439b0 con 0x7f2a6006c820 2023-12-01T17:36:32.160 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 0 to deep-scrub 2023-12-01T17:36:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 10 librados: watch_flush enter 2023-12-01T17:36:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 10 librados: watch_flush exit 2023-12-01T17:36:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 20 client.23167.objecter shutdown clearing up homeless session... 2023-12-01T17:36:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 10 client.23167.objecter successfully canceled tick 2023-12-01T17:36:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2a6006c820 msgr2=0x7f2a6006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2a6006c820 0x7f2a6006ece0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f2a78007fd0 tx=0x7f2a78007e80 comp rx=0 tx=0).stop 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 10 monclient: shutdown 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a7c1423b0 msgr2=0x7f2a7c145a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a7c1423b0 0x7f2a7c145a60 secure :-1 s=READY pgs=5886 cs=0 l=1 rev1=1 crypto rx=0x7f2a74009eb0 tx=0x7f2a74004a40 comp rx=0 tx=0).stop 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 shutdown_connections 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2a7c1423b0 0x7f2a7c145a60 unknown :-1 s=CLOSED pgs=5886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a7c0a8e00 0x7f2a7c145520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f2a6006c820 0x7f2a6006ece0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 --2- 172.21.15.67:0/832059656 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f2a7c0a8a20 0x7f2a7c138190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.163+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 >> 172.21.15.67:0/832059656 conn(0x7f2a7c0b7870 msgr2=0x7f2a7c13e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.164+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 shutdown_connections 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.164+0000 7f2a83689640 1 -- 172.21.15.67:0/832059656 wait complete. 2023-12-01T17:36:32.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.164+0000 7f2a83689640 1 librados: shutdown 2023-12-01T17:36:32.174 INFO:tasks.ceph:Scrubbing osd.1 2023-12-01T17:36:32.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:32.284 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.285+0000 7f7184b89640 10 monclient: get_monmap_and_config 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.285+0000 7f7184b89640 10 monclient: build_initial_monmap 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.285+0000 7f7184b89640 10 monclient: monmap: 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:32.286297+0000 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:32.286297+0000 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:32.285 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:32.286 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:32.286 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 1 Processor -- start 2023-12-01T17:36:32.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 1 -- start start 2023-12-01T17:36:32.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient: init 2023-12-01T17:36:32.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:32.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-12-01T17:36:32.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f71800a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:32.287 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f71800a8e00 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f718013f5c0 0x7f7180142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient(hunting): picked mon.noname-a con 0x7f718013f5c0 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71800b21f0 con 0x7f71800a8a20 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71800a9ed0 con 0x7f71800a8e00 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71800a9cc0 con 0x7f718013f5c0 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.286+0000 7f7184b89640 10 monclient(hunting): authenticate will time out at 3323.524360s 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.287+0000 7f717ed9d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f718013f5c0 0x7f7180142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.287+0000 7f717dd9b640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f71800a8e00 0x7f71801423b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:36928/0 (socket says 172.21.15.67:36928) 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.287+0000 7f717dd9b640 1 -- 172.21.15.67:0/3191267186 learned_addr learned my addr 172.21.15.67:0/3191267186 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:32.288 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.288+0000 7f717ed9d640 10 monclient(hunting): get_auth_request con 0x7f718013f5c0 auth_method 0 2023-12-01T17:36:32.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.288+0000 7f717ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.288+0000 7f717ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.289 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.288+0000 7f717ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.288+0000 7f717ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.288+0000 7f717ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.288+0000 7f717ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.289+0000 7f717d59a640 1 -- 172.21.15.67:0/3191267186 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442651072 0 0) 0x7f71800a9ed0 con 0x7f71800a8e00 2023-12-01T17:36:32.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.289+0000 7f717d59a640 10 monclient(hunting): _negotiate 2023-12-01T17:36:32.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.289+0000 7f717d59a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.289+0000 7f717d59a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.289+0000 7f717d59a640 10 monclient(hunting): my global_id is 23170 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.289+0000 7f717d59a640 1 -- 172.21.15.67:0/3191267186 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7168003520 con 0x7f71800a8e00 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 10 monclient(hunting): handle_auth_done global_id 15972 payload 274 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 1 -- 172.21.15.67:0/3191267186 >> v1:172.21.15.92:6789/0 conn(0x7f71800a8e00 legacy=0x7f71801423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 1 -- 172.21.15.67:0/3191267186 >> v1:172.21.15.67:6789/0 conn(0x7f71800a8a20 legacy=0x7f718013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 10 monclient: _finish_hunting 0 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 1 monclient: found mon.noname-a 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:36:32.291 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.291+0000 7f717ed9d640 1 -- 172.21.15.67:0/3191267186 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71800aac70 con 0x7f718013f5c0 2023-12-01T17:36:32.294 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.294+0000 7f717ed9d640 1 --2- 172.21.15.67:0/3191267186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 0x7f7180142ac0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7f7178009d90 tx=0x7f717802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c70c335b47d1c5f7 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 1 -- 172.21.15.67:0/3191267186 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802cbd0 con 0x7f718013f5c0 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: dump: 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.299 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: _finish_auth 0 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: _check_auth_tickets 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 1 -- 172.21.15.67:0/3191267186 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f717802cd30 con 0x7f718013f5c0 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f7184b89640 5 monclient: authenticate success, global_id 15972 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f7184b89640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f7184b89640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:32.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f7184b89640 4 set_mon_vals no callback set 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 1 -- 172.21.15.67:0/3191267186 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f717802cea0 con 0x7f718013f5c0 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.299+0000 7f717d59a640 10 monclient: dump: 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.301 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.302 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.305 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 10 monclient: shutdown 2023-12-01T17:36:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 1 -- 172.21.15.67:0/3191267186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 msgr2=0x7f7180142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.306 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 1 --2- 172.21.15.67:0/3191267186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 0x7f7180142ac0 secure :-1 s=READY pgs=3166 cs=0 l=1 rev1=1 crypto rx=0x7f7178009d90 tx=0x7f717802be20 comp rx=0 tx=0).stop 2023-12-01T17:36:32.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 1 -- 172.21.15.67:0/3191267186 shutdown_connections 2023-12-01T17:36:32.307 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 1 --2- 172.21.15.67:0/3191267186 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 0x7f7180142ac0 unknown :-1 s=CLOSED pgs=3166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 1 -- 172.21.15.67:0/3191267186 >> 172.21.15.67:0/3191267186 conn(0x7f71800b7870 msgr2=0x7f71800b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 1 -- 172.21.15.67:0/3191267186 shutdown_connections 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.300+0000 7f7184b89640 1 -- 172.21.15.67:0/3191267186 wait complete. 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 10 monclient: build_initial_monmap 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 10 monclient: monmap: 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:32.308 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 1 librados: starting msgr at 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 1 librados: starting objecter 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 1 Processor -- start 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 1 -- start start 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 1 librados: setting wanted keys 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 1 librados: calling monclient init 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.301+0000 7f7184b89640 10 monclient: init 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71800a8a20 0x7f71801399d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): picked mon.noname-b con 0x7f71800a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71800a8e00 0x7f718013a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): picked mon.noname-c con 0x7f71800a8e00 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 0x7f718013a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.309 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): picked mon.noname-a con 0x7f718013f5c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f718013b590 con 0x7f718013f5c0 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71800ab5c0 con 0x7f71800a8a20 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71800ab8a0 con 0x7f71800a8e00 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.302+0000 7f7184b89640 10 monclient(hunting): authenticate will time out at 3323.539803s 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 0x7f718013a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71800a8a20 0x7f71801399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71800a8e00 0x7f718013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71800a8a20 0x7f71801399d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42608/0 (socket says 172.21.15.67:42608) 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 1 -- 172.21.15.67:0/1523623590 learned_addr learned my addr 172.21.15.67:0/1523623590 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 10 monclient(hunting): get_auth_request con 0x7f71800a8a20 auth_method 0 2023-12-01T17:36:32.310 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 0x7f718013a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37802/0 (socket says 172.21.15.67:37802) 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 10 monclient(hunting): get_auth_request con 0x7f718013f5c0 auth_method 0 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 10 monclient(hunting): get_auth_request con 0x7f71800a8e00 auth_method 0 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.311 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.303+0000 7f717dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 10 monclient(hunting): handle_auth_done global_id 23173 payload 824 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 1 -- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71800a8e00 msgr2=0x7f718013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f71800a8e00 0x7f718013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 1 -- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 msgr2=0x7f718013a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f718013f5c0 0x7f718013a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 10 monclient: _finish_hunting 0 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 1 monclient: found mon.noname-b 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.306+0000 7f717e59c640 1 -- 172.21.15.67:0/1523623590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7178009a70 con 0x7f71800a8a20 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.308+0000 7f717e59c640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71800a8a20 0x7f71801399d0 secure :-1 s=READY pgs=5888 cs=0 l=1 rev1=1 crypto rx=0x7f717000c400 tx=0x7f717000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.308+0000 7f716effd640 10 client.?.objecter ms_handle_connect 0x7f71800a8a20 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.308+0000 7f716effd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71700052d0 con 0x7f71800a8a20 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 10 monclient: dump: 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.314 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 10 monclient: _finish_auth 0 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.311+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7170005450 con 0x7f71800a8a20 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 5 monclient: authenticate success, global_id 23173 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 monclient: _renew_subs 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f718013adc0 con 0x7f71800a8a20 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f716effd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:32.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170007260 con 0x7f71800a8a20 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f716effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f716effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f716effd640 10 monclient: dump: 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.316 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f717cd99640 4 set_mon_vals no callback set 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 client.23173.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 monclient: _renew_subs 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71800ac180 con 0x7f71800a8a20 2023-12-01T17:36:32.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 1 librados: init done 2023-12-01T17:36:32.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 client.23173.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:32.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 15 client.23173.objecter _session_command_op_assign -1 1 2023-12-01T17:36:32.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 20 client.23173.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:32.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 client.23173.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:32.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.318 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.312+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f718013fb50 con 0x7f71800a8a20 2023-12-01T17:36:32.331 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.332+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f71700073c0 con 0x7f71800a8a20 2023-12-01T17:36:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.332+0000 7f716effd640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f716006c870 0x7f716006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.332+0000 7f717dd9b640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f716006c870 0x7f716006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.332+0000 7f717dd9b640 10 monclient: get_auth_request con 0x7f716006c870 auth_method 0 2023-12-01T17:36:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.332+0000 7f717dd9b640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f716006c870 0x7f716006ed30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f71800b7d50 tx=0x7f7174009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:32.332 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.333+0000 7f716effd640 10 client.23173.objecter ms_handle_connect 0x7f716006c870 2023-12-01T17:36:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.333+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f7170007670 con 0x7f71800a8a20 2023-12-01T17:36:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.333+0000 7f716effd640 10 client.23173.objecter ms_dispatch 0x7f7180000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.333+0000 7f716effd640 3 client.23173.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.333+0000 7f716effd640 3 client.23173.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.333+0000 7f716effd640 10 client.23173.objecter checking command 1 2023-12-01T17:36:32.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f71600743d0 0x7f7160076870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 20 client.23173.objecter _get_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 20 client.23173.objecter put_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 15 client.23173.objecter _session_command_op_remove -1 1 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 20 client.23173.objecter _get_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 20 client.23173.objecter get_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 15 client.23173.objecter _session_command_op_assign 1 1 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 20 client.23173.objecter put_session s=0x7f7160073d00 osd=1 4 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 10 client.23173.objecter _send_command 1 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 --> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7160076f40 con 0x7f71600743d0 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 20 client.23173.objecter dump_active .. 0 homeless 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=1099) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71700545f0 con 0x7f71800a8a20 2023-12-01T17:36:32.334 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f716effd640 10 monclient: handle_get_version_reply finishing 1 version 1099 2023-12-01T17:36:32.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f717ed9d640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f71600743d0 0x7f7160076870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.334+0000 7f717ed9d640 10 monclient: get_auth_request con 0x7f71600743d0 auth_method 0 2023-12-01T17:36:32.336 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.337+0000 7f716effd640 10 client.23173.objecter ms_handle_connect 0x7f71600743d0 2023-12-01T17:36:32.336 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.337+0000 7f717ed9d640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f71600743d0 0x7f7160076870 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:32.362 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.363+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== osd.1 v2:172.21.15.67:6826/866081726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f7160076f40 con 0x7f71600743d0 2023-12-01T17:36:32.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.363+0000 7f716effd640 10 client.23173.objecter ms_dispatch 0x7f7180000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:32.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.363+0000 7f716effd640 10 client.23173.objecter _finish_command 1 = system:0 2023-12-01T17:36:32.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.363+0000 7f716effd640 20 client.23173.objecter put_session s=0x7f7160073d00 osd=1 4 2023-12-01T17:36:32.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.363+0000 7f716effd640 15 client.23173.objecter _session_command_op_remove 1 1 2023-12-01T17:36:32.378 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 10 client.23173.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 15 client.23173.objecter _session_command_op_assign -1 2 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 20 client.23173.objecter _get_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 20 client.23173.objecter put_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 20 client.23173.objecter _get_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 15 client.23173.objecter _session_command_op_remove -1 2 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 20 client.23173.objecter get_session s=0x7f7160073d00 osd=1 3 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 15 client.23173.objecter _session_command_op_assign 1 2 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 20 client.23173.objecter put_session s=0x7f7160073d00 osd=1 4 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 10 client.23173.objecter _send_command 2 2023-12-01T17:36:32.379 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.376+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 --> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f71800b3f40 con 0x7f71600743d0 2023-12-01T17:36:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f716effd640 1 -- 172.21.15.67:0/1523623590 <== osd.1 v2:172.21.15.67:6826/866081726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f71800b3f40 con 0x7f71600743d0 2023-12-01T17:36:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f716effd640 10 client.23173.objecter ms_dispatch 0x7f7180000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f716effd640 10 client.23173.objecter _finish_command 2 = system:0 2023-12-01T17:36:32.389 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f716effd640 20 client.23173.objecter put_session s=0x7f7160073d00 osd=1 4 2023-12-01T17:36:32.390 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f716effd640 15 client.23173.objecter _session_command_op_remove 1 2 2023-12-01T17:36:32.390 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:32.390 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:32.390 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 10 librados: watch_flush enter 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 10 librados: watch_flush exit 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 10 client.23173.objecter close_session for osd.1 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f71600743d0 msgr2=0x7f7160076870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6826/866081726,v1:172.21.15.67:6827/866081726] conn(0x7f71600743d0 0x7f7160076870 crc :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 20 client.23173.objecter put_session s=0x7f7160073d00 osd=1 1 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 20 client.23173.objecter shutdown clearing up homeless session... 2023-12-01T17:36:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 10 client.23173.objecter successfully canceled tick 2023-12-01T17:36:32.393 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f716006c870 msgr2=0x7f716006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.393 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f716006c870 0x7f716006ed30 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f71800b7d50 tx=0x7f7174009040 comp rx=0 tx=0).stop 2023-12-01T17:36:32.393 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 10 monclient: shutdown 2023-12-01T17:36:32.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:32.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71800a8a20 msgr2=0x7f71801399d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.394 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 --2- 172.21.15.67:0/1523623590 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f71800a8a20 0x7f71801399d0 secure :-1 s=READY pgs=5888 cs=0 l=1 rev1=1 crypto rx=0x7f717000c400 tx=0x7f717000c8c0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f717ed9d640 1 -- 172.21.15.67:0/1523623590 reap_dead start 2023-12-01T17:36:32.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 shutdown_connections 2023-12-01T17:36:32.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 >> 172.21.15.67:0/1523623590 conn(0x7f71800b7870 msgr2=0x7f718013eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:32.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 shutdown_connections 2023-12-01T17:36:32.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 -- 172.21.15.67:0/1523623590 wait complete. 2023-12-01T17:36:32.395 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.390+0000 7f7184b89640 1 librados: shutdown 2023-12-01T17:36:32.400 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-12-01T17:36:32.520 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.520+0000 7ff2ef533640 10 monclient: get_monmap_and_config 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.520+0000 7ff2ef533640 10 monclient: build_initial_monmap 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.520+0000 7ff2ef533640 10 monclient: monmap: 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:32.521388+0000 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:32.521388+0000 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:32.521 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.521+0000 7ff2ef533640 1 Processor -- start 2023-12-01T17:36:32.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.521+0000 7ff2ef533640 1 -- start start 2023-12-01T17:36:32.522 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.521+0000 7ff2ef533640 10 monclient: init 2023-12-01T17:36:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-12-01T17:36:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff2e80a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:32.525 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e81461c0 0x7ff2e80a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient(hunting): picked mon.noname-b con 0x7ff2e81461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:32.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2e813f0f0 0x7ff2e813f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient(hunting): picked mon.noname-a con 0x7ff2e813f0f0 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2e8145620 con 0x7ff2e80a8a20 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2e80aace0 con 0x7ff2e813f0f0 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e80aafc0 con 0x7ff2e81461c0 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ef533640 10 monclient(hunting): authenticate will time out at 3323.759520s 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2edad0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff2e813f0f0 0x7ff2e813f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.522+0000 7ff2ecace640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e81461c0 0x7ff2e80a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e81461c0 0x7ff2e80a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:45718/0 (socket says 172.21.15.67:45718) 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 1 -- 172.21.15.67:0/4160512974 learned_addr learned my addr 172.21.15.67:0/4160512974 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 10 monclient(hunting): get_auth_request con 0x7ff2e81461c0 auth_method 0 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2edad0640 10 monclient(hunting): get_auth_request con 0x7ff2e813f0f0 auth_method 0 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2edad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2edad0640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2edad0640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.523+0000 7ff2ecace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.528 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.524+0000 7ff2edad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.524+0000 7ff2edad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.524+0000 7ff2edad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 10 monclient(hunting): handle_auth_done global_id 17927 payload 274 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 1 -- 172.21.15.67:0/4160512974 >> v2:172.21.15.67:3300/0 conn(0x7ff2e813f0f0 msgr2=0x7ff2e813f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 1 --2- 172.21.15.67:0/4160512974 >> v2:172.21.15.67:3300/0 conn(0x7ff2e813f0f0 0x7ff2e813f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 1 -- 172.21.15.67:0/4160512974 >> v1:172.21.15.67:6789/0 conn(0x7ff2e80a8a20 legacy=0x7ff2e8145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 10 monclient: _finish_hunting 0 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 1 monclient: found mon.noname-b 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:32.529 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.525+0000 7ff2ecace640 1 -- 172.21.15.67:0/4160512974 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e813fac0 con 0x7ff2e81461c0 2023-12-01T17:36:32.530 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.528+0000 7ff2ecace640 1 --2- 172.21.15.67:0/4160512974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e81461c0 0x7ff2e80a8e00 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009870 tx=0x7ff2d802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0ca665f2ce7fd96 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 1 -- 172.21.15.67:0/4160512974 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8030840 con 0x7ff2e81461c0 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: dump: 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.532 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: _finish_auth 0 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 1 -- 172.21.15.67:0/4160512974 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d80309a0 con 0x7ff2e81461c0 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2ef533640 5 monclient: authenticate success, global_id 17927 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2ef533640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2ef533640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2ef533640 4 set_mon_vals no callback set 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 1 -- 172.21.15.67:0/4160512974 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8030d30 con 0x7ff2e81461c0 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.532+0000 7ff2dffff640 10 monclient: dump: 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.535 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.536 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.536 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 10 monclient: shutdown 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 -- 172.21.15.67:0/4160512974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e81461c0 msgr2=0x7ff2e80a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 --2- 172.21.15.67:0/4160512974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e81461c0 0x7ff2e80a8e00 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8009870 tx=0x7ff2d802b980 comp rx=0 tx=0).stop 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 -- 172.21.15.67:0/4160512974 shutdown_connections 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 --2- 172.21.15.67:0/4160512974 >> v2:172.21.15.67:3300/0 conn(0x7ff2e813f0f0 0x7ff2e813f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 --2- 172.21.15.67:0/4160512974 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e81461c0 0x7ff2e80a8e00 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 -- 172.21.15.67:0/4160512974 >> 172.21.15.67:0/4160512974 conn(0x7ff2e80b7890 msgr2=0x7ff2e80b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 -- 172.21.15.67:0/4160512974 shutdown_connections 2023-12-01T17:36:32.540 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.533+0000 7ff2ef533640 1 -- 172.21.15.67:0/4160512974 wait complete. 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 10 monclient: build_initial_monmap 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 10 monclient: monmap: 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 1 librados: starting msgr at 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 1 librados: starting objecter 2023-12-01T17:36:32.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 1 Processor -- start 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 1 -- start start 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 1 librados: setting wanted keys 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 1 librados: calling monclient init 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.534+0000 7ff2ef533640 10 monclient: init 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient: _add_conns ranks=[2,1,0] 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e80a8a20 0x7ff2e80b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): picked mon.noname-c con 0x7ff2e80a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e813f0f0 0x7ff2e8138e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): picked mon.noname-b con 0x7ff2e813f0f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81461c0 0x7ff2e81da570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): picked mon.noname-a con 0x7ff2e81461c0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:32.542 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e813ffe0 con 0x7ff2e81461c0 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e8141d00 con 0x7ff2e813f0f0 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e81393a0 con 0x7ff2e80a8a20 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ef533640 10 monclient(hunting): authenticate will time out at 3323.772553s 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2edad0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81461c0 0x7ff2e81da570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ed2cf640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e80a8a20 0x7ff2e80b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.535+0000 7ff2ecace640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e813f0f0 0x7ff2e8138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e80a8a20 0x7ff2e80b70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:45732/0 (socket says 172.21.15.67:45732) 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81461c0 0x7ff2e81da570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37828/0 (socket says 172.21.15.67:37828) 2023-12-01T17:36:32.543 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 1 -- 172.21.15.67:0/1692019509 learned_addr learned my addr 172.21.15.67:0/1692019509 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 10 monclient(hunting): get_auth_request con 0x7ff2e80a8a20 auth_method 0 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 10 monclient(hunting): get_auth_request con 0x7ff2e81461c0 auth_method 0 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.544 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ecace640 10 monclient(hunting): get_auth_request con 0x7ff2e813f0f0 auth_method 0 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ecace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ecace640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ecace640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ed2cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2edad0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ecace640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ecace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.536+0000 7ff2ecace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 10 monclient(hunting): handle_auth_done global_id 23176 payload 824 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 1 -- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e80a8a20 msgr2=0x7ff2e80b70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e80a8a20 0x7ff2e80b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 1 -- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81461c0 msgr2=0x7ff2e81da570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81461c0 0x7ff2e81da570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 10 monclient: _finish_hunting 0 2023-12-01T17:36:32.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 1 monclient: found mon.noname-b 2023-12-01T17:36:32.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:32.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.539+0000 7ff2ecace640 1 -- 172.21.15.67:0/1692019509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d80094d0 con 0x7ff2e813f0f0 2023-12-01T17:36:32.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.541+0000 7ff2ecace640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e813f0f0 0x7ff2e8138e60 secure :-1 s=READY pgs=5889 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8033000 tx=0x7ff2d8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.541+0000 7ff2ddffb640 10 client.?.objecter ms_handle_connect 0x7ff2e813f0f0 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.541+0000 7ff2ddffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 1 -- 172.21.15.67:0/1692019509 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8039a90 con 0x7ff2e813f0f0 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 10 monclient: dump: 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.548 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 1 -- 172.21.15.67:0/1692019509 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d8039c10 con 0x7ff2e813f0f0 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ddffb640 1 -- 172.21.15.67:0/1692019509 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d8043410 con 0x7ff2e813f0f0 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ef533640 5 monclient: authenticate success, global_id 23176 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ef533640 10 monclient: _renew_subs 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.544+0000 7ff2ef533640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e8140470 con 0x7ff2e813f0f0 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:32.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ddffb640 10 monclient: dump: 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2df7fe640 4 set_mon_vals no callback set 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ef533640 10 client.23176.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:32.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ef533640 10 monclient: _renew_subs 2023-12-01T17:36:32.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ef533640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e81dac90 con 0x7ff2e813f0f0 2023-12-01T17:36:32.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.545+0000 7ff2ef533640 1 librados: init done 2023-12-01T17:36:32.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.546+0000 7ff2ef533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:32.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.546+0000 7ff2ef533640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:32.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.546+0000 7ff2ef533640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:32.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.546+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e813fdb0 con 0x7ff2e813f0f0 2023-12-01T17:36:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.565+0000 7ff2ddffb640 1 -- 172.21.15.67:0/1692019509 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7ff2d804c3b0 con 0x7ff2e813f0f0 2023-12-01T17:36:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.566+0000 7ff2ddffb640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7ff2cc06c870 0x7ff2cc06ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:32.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.566+0000 7ff2ed2cf640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7ff2cc06c870 0x7ff2cc06ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.566+0000 7ff2ed2cf640 10 monclient: get_auth_request con 0x7ff2cc06c870 auth_method 0 2023-12-01T17:36:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.566+0000 7ff2ed2cf640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7ff2cc06c870 0x7ff2cc06ed30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4001840 tx=0x7ff2e4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:32.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.566+0000 7ff2ddffb640 10 client.23176.objecter ms_handle_connect 0x7ff2cc06c870 2023-12-01T17:36:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.789+0000 7ff2ddffb640 1 -- 172.21.15.67:0/1692019509 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7ff2d807a890 con 0x7ff2e813f0f0 2023-12-01T17:36:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.789+0000 7ff2ddffb640 10 client.23176.objecter ms_dispatch 0x7ff2e8000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.789+0000 7ff2ddffb640 3 client.23176.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.789+0000 7ff2ddffb640 3 client.23176.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:32.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.789+0000 7ff2ddffb640 20 client.23176.objecter dump_active .. 0 homeless 2023-12-01T17:36:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.798+0000 7ff2ddffb640 1 -- 172.21.15.67:0/1692019509 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2d8076bd0 con 0x7ff2e813f0f0 2023-12-01T17:36:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.798+0000 7ff2ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:32.798 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.798+0000 7ff2ddffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:32.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.979+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff2e814c880 con 0x7ff2cc06c870 2023-12-01T17:36:32.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.979+0000 7ff2ddffb640 1 -- 172.21.15.67:0/1692019509 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff2e40014e0 con 0x7ff2cc06c870 2023-12-01T17:36:32.979 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 1 to deep-scrub 2023-12-01T17:36:32.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 10 librados: watch_flush enter 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 10 librados: watch_flush exit 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 20 client.23176.objecter shutdown clearing up homeless session... 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 10 client.23176.objecter successfully canceled tick 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7ff2cc06c870 msgr2=0x7ff2cc06ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7ff2cc06c870 0x7ff2cc06ed30 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff2e4001840 tx=0x7ff2e4009040 comp rx=0 tx=0).stop 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 10 monclient: shutdown 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e813f0f0 msgr2=0x7ff2e8138e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e813f0f0 0x7ff2e8138e60 secure :-1 s=READY pgs=5889 cs=0 l=1 rev1=1 crypto rx=0x7ff2d8033000 tx=0x7ff2d8004730 comp rx=0 tx=0).stop 2023-12-01T17:36:32.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 shutdown_connections 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2e81461c0 0x7ff2e81da570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff2e813f0f0 0x7ff2e8138e60 unknown :-1 s=CLOSED pgs=5889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7ff2cc06c870 0x7ff2cc06ed30 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 --2- 172.21.15.67:0/1692019509 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7ff2e80a8a20 0x7ff2e80b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 >> 172.21.15.67:0/1692019509 conn(0x7ff2e80b7890 msgr2=0x7ff2e8139770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 shutdown_connections 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 -- 172.21.15.67:0/1692019509 wait complete. 2023-12-01T17:36:32.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:32.981+0000 7ff2ef533640 1 librados: shutdown 2023-12-01T17:36:32.993 INFO:tasks.ceph:Scrubbing osd.2 2023-12-01T17:36:32.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.103+0000 7f8babeb0640 10 monclient: get_monmap_and_config 2023-12-01T17:36:33.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.103+0000 7f8babeb0640 10 monclient: build_initial_monmap 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.103+0000 7f8babeb0640 10 monclient: monmap: 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:33.104090+0000 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:33.104090+0000 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:33.103 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.103+0000 7f8babeb0640 1 Processor -- start 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 1 -- start start 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient: init 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ba40a8a20 0x7f8ba413eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba40a8a20 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8ba40a8e00 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8ba41423b0 0x7f8ba4142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba41423b0 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ba40b21f0 con 0x7f8ba40a8e00 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba40a9cc0 con 0x7f8ba40a8a20 2023-12-01T17:36:33.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba40a9ed0 con 0x7f8ba41423b0 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.104+0000 7f8babeb0640 10 monclient(hunting): authenticate will time out at 3324.342145s 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8ba9c4c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ba40a8a20 0x7f8ba413eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8ba9c4c640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ba40a8a20 0x7f8ba413eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37830/0 (socket says 172.21.15.67:37830) 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8ba9c4c640 1 -- 172.21.15.67:0/577682344 learned_addr learned my addr 172.21.15.67:0/577682344 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8ba9c4c640 10 monclient(hunting): get_auth_request con 0x7f8ba40a8a20 auth_method 0 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8ba9c4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8ba9c4c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8ba9c4c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.105+0000 7f8baa44d640 1 --2- 172.21.15.67:0/577682344 >> v2:172.21.15.92:3300/0 conn(0x7f8ba41423b0 0x7f8ba4142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.106+0000 7f8baa44d640 10 monclient(hunting): get_auth_request con 0x7f8ba41423b0 auth_method 0 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.106+0000 7f8baa44d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.106+0000 7f8baa44d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.106+0000 7f8baa44d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.106+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.106+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.106+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.107+0000 7f8baa44d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.107+0000 7f8baa44d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.107+0000 7f8baa44d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.107+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_done global_id 15990 payload 274 2023-12-01T17:36:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.107+0000 7f8ba9c4c640 1 -- 172.21.15.67:0/577682344 >> v2:172.21.15.92:3300/0 conn(0x7f8ba41423b0 msgr2=0x7f8ba4142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.107+0000 7f8ba9c4c640 1 --2- 172.21.15.67:0/577682344 >> v2:172.21.15.92:3300/0 conn(0x7f8ba41423b0 0x7f8ba4142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.108+0000 7f8ba9c4c640 1 -- 172.21.15.67:0/577682344 >> v1:172.21.15.92:6789/0 conn(0x7f8ba40a8e00 legacy=0x7f8ba413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.108+0000 7f8ba9c4c640 10 monclient: _finish_hunting 0 2023-12-01T17:36:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.108+0000 7f8ba9c4c640 1 monclient: found mon.noname-a 2023-12-01T17:36:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.108+0000 7f8ba9c4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.108+0000 7f8ba9c4c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:36:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.108+0000 7f8ba9c4c640 1 -- 172.21.15.67:0/577682344 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba40aac70 con 0x7f8ba40a8a20 2023-12-01T17:36:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.109+0000 7f8ba9c4c640 1 --2- 172.21.15.67:0/577682344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8a20 0x7f8ba413eeb0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40b39a0 tx=0x7f8b9c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bbfda099212f63 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 1 -- 172.21.15.67:0/577682344 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c0317e0 con 0x7f8ba40a8a20 2023-12-01T17:36:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:36:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: dump: 2023-12-01T17:36:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.113 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: _finish_auth 0 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: _check_auth_tickets 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 1 -- 172.21.15.67:0/577682344 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c031940 con 0x7f8ba40a8a20 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 1 -- 172.21.15.67:0/577682344 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c031cd0 con 0x7f8ba40a8a20 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8ba8c4a640 10 monclient: dump: 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8babeb0640 5 monclient: authenticate success, global_id 15990 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8babeb0640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:33.115 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.113+0000 7f8babeb0640 4 set_mon_vals no callback set 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 10 monclient: shutdown 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 -- 172.21.15.67:0/577682344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8a20 msgr2=0x7f8ba413eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 --2- 172.21.15.67:0/577682344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8a20 0x7f8ba413eeb0 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f8ba40b39a0 tx=0x7f8b9c02b880 comp rx=0 tx=0).stop 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 -- 172.21.15.67:0/577682344 shutdown_connections 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 --2- 172.21.15.67:0/577682344 >> v2:172.21.15.92:3300/0 conn(0x7f8ba41423b0 0x7f8ba4142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 --2- 172.21.15.67:0/577682344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8a20 0x7f8ba413eeb0 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 -- 172.21.15.67:0/577682344 >> 172.21.15.67:0/577682344 conn(0x7f8ba40b7870 msgr2=0x7f8ba40b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 -- 172.21.15.67:0/577682344 shutdown_connections 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.114+0000 7f8babeb0640 1 -- 172.21.15.67:0/577682344 wait complete. 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 10 monclient: build_initial_monmap 2023-12-01T17:36:33.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 10 monclient: monmap: 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 1 librados: starting msgr at 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 1 librados: starting objecter 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 1 Processor -- start 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 1 -- start start 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 1 librados: setting wanted keys 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 1 librados: calling monclient init 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.115+0000 7f8babeb0640 10 monclient: init 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ba40a8a20 0x7f8ba4138f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): picked mon.noname-b con 0x7f8ba40a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8e00 0x7f8ba413aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): picked mon.noname-a con 0x7f8ba40a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ba41423b0 0x7f8ba41d76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): picked mon.noname-c con 0x7f8ba41423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba41d6ef0 con 0x7f8ba40a8e00 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba41d71d0 con 0x7f8ba40a8a20 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba41d7c30 con 0x7f8ba41423b0 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.116+0000 7f8babeb0640 10 monclient(hunting): authenticate will time out at 3324.353829s 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8baa44d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ba41423b0 0x7f8ba41d76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8e00 0x7f8ba413aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ba40a8a20 0x7f8ba4138f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8e00 0x7f8ba413aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37836/0 (socket says 172.21.15.67:37836) 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 1 -- 172.21.15.67:0/4099324449 learned_addr learned my addr 172.21.15.67:0/4099324449 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 10 monclient(hunting): get_auth_request con 0x7f8ba40a8e00 auth_method 0 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 10 monclient(hunting): get_auth_request con 0x7f8ba40a8a20 auth_method 0 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8baa44d640 10 monclient(hunting): get_auth_request con 0x7f8ba41423b0 auth_method 0 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8baa44d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8baa44d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8baa44d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.117+0000 7f8ba944b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.118+0000 7f8baa44d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.118+0000 7f8baa44d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.118+0000 7f8baa44d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 10 monclient(hunting): handle_auth_done global_id 23182 payload 824 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 1 -- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ba41423b0 msgr2=0x7f8ba41d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ba41423b0 0x7f8ba41d76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 1 -- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8e00 msgr2=0x7f8ba413aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ba40a8e00 0x7f8ba413aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 10 monclient: _finish_hunting 0 2023-12-01T17:36:33.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 1 monclient: found mon.noname-b 2023-12-01T17:36:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.120+0000 7f8ba9c4c640 1 -- 172.21.15.67:0/4099324449 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c0094d0 con 0x7f8ba40a8a20 2023-12-01T17:36:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.122+0000 7f8ba9c4c640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ba40a8a20 0x7f8ba4138f80 secure :-1 s=READY pgs=5891 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c033000 tx=0x7f8b9c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.122+0000 7f8ba27fc640 10 client.?.objecter ms_handle_connect 0x7f8ba40a8a20 2023-12-01T17:36:33.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.122+0000 7f8ba27fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c032880 con 0x7f8ba40a8a20 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: dump: 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.128 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: _finish_auth 0 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: _check_auth_tickets 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c0329e0 con 0x7f8ba40a8a20 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 5 monclient: authenticate success, global_id 23182 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 monclient: _renew_subs 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba41456d0 con 0x7f8ba40a8a20 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:33.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c032d90 con 0x7f8ba40a8a20 2023-12-01T17:36:33.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:33.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba27fc640 10 monclient: dump: 2023-12-01T17:36:33.130 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.130 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.130 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8ba3fff640 4 set_mon_vals no callback set 2023-12-01T17:36:33.131 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 client.23182.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 monclient: _renew_subs 2023-12-01T17:36:33.132 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba4145970 con 0x7f8ba40a8a20 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 1 librados: init done 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 client.23182.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 15 client.23182.objecter _session_command_op_assign -1 1 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 20 client.23182.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 client.23182.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.126+0000 7f8babeb0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.133 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.127+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ba41d7060 con 0x7f8ba40a8a20 2023-12-01T17:36:33.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.145+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8b9c04e450 con 0x7f8ba40a8a20 2023-12-01T17:36:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.145+0000 7f8ba27fc640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c870 0x7f8b9006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.145+0000 7f8ba944b640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c870 0x7f8b9006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.145+0000 7f8ba944b640 10 monclient: get_auth_request con 0x7f8b9006c870 auth_method 0 2023-12-01T17:36:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.145+0000 7f8ba27fc640 10 client.23182.objecter ms_handle_connect 0x7f8b9006c870 2023-12-01T17:36:33.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.145+0000 7f8ba944b640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c870 0x7f8b9006ed30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8ba413c800 tx=0x7f8b98009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f8b9c07a700 con 0x7f8ba40a8a20 2023-12-01T17:36:33.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 10 client.23182.objecter ms_dispatch 0x7f8ba4000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:33.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 3 client.23182.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:33.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 3 client.23182.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:33.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 10 client.23182.objecter checking command 1 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7f8b90074460 0x7f8b90076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 20 client.23182.objecter _get_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 20 client.23182.objecter put_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 15 client.23182.objecter _session_command_op_remove -1 1 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 20 client.23182.objecter _get_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 20 client.23182.objecter get_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 15 client.23182.objecter _session_command_op_assign 2 1 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 20 client.23182.objecter put_session s=0x7f8b90073d90 osd=2 4 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 10 client.23182.objecter _send_command 1 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b90076fd0 con 0x7f8b90074460 2023-12-01T17:36:33.147 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8ba27fc640 20 client.23182.objecter dump_active .. 0 homeless 2023-12-01T17:36:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8baa44d640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7f8b90074460 0x7f8b90076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.147+0000 7f8baa44d640 10 monclient: get_auth_request con 0x7f8b90074460 auth_method 0 2023-12-01T17:36:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.148+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=1099) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b9c076a40 con 0x7f8ba40a8a20 2023-12-01T17:36:33.148 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.148+0000 7f8ba27fc640 10 monclient: handle_get_version_reply finishing 1 version 1099 2023-12-01T17:36:33.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.150+0000 7f8baa44d640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7f8b90074460 0x7f8b90076900 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.150+0000 7f8ba27fc640 10 client.23182.objecter ms_handle_connect 0x7f8b90074460 2023-12-01T17:36:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.175+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== osd.2 v2:172.21.15.67:6818/2342127620 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8b90076fd0 con 0x7f8b90074460 2023-12-01T17:36:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.175+0000 7f8ba27fc640 10 client.23182.objecter ms_dispatch 0x7f8ba4000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.175+0000 7f8ba27fc640 10 client.23182.objecter _finish_command 1 = system:0 2023-12-01T17:36:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.175+0000 7f8ba27fc640 20 client.23182.objecter put_session s=0x7f8b90073d90 osd=2 4 2023-12-01T17:36:33.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.175+0000 7f8ba27fc640 15 client.23182.objecter _session_command_op_remove 2 1 2023-12-01T17:36:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 10 client.23182.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 15 client.23182.objecter _session_command_op_assign -1 2 2023-12-01T17:36:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 20 client.23182.objecter _get_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 20 client.23182.objecter put_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 20 client.23182.objecter _get_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.188 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 15 client.23182.objecter _session_command_op_remove -1 2 2023-12-01T17:36:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 20 client.23182.objecter get_session s=0x7f8b90073d90 osd=2 3 2023-12-01T17:36:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 15 client.23182.objecter _session_command_op_assign 2 2 2023-12-01T17:36:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 20 client.23182.objecter put_session s=0x7f8b90073d90 osd=2 4 2023-12-01T17:36:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 10 client.23182.objecter _send_command 2 2023-12-01T17:36:33.189 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.188+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 --> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8ba40b1ac0 con 0x7f8b90074460 2023-12-01T17:36:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8ba27fc640 1 -- 172.21.15.67:0/4099324449 <== osd.2 v2:172.21.15.67:6818/2342127620 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8ba40b1ac0 con 0x7f8b90074460 2023-12-01T17:36:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8ba27fc640 10 client.23182.objecter ms_dispatch 0x7f8ba4000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8ba27fc640 10 client.23182.objecter _finish_command 2 = system:0 2023-12-01T17:36:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8ba27fc640 20 client.23182.objecter put_session s=0x7f8b90073d90 osd=2 4 2023-12-01T17:36:33.202 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8ba27fc640 15 client.23182.objecter _session_command_op_remove 2 2 2023-12-01T17:36:33.202 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:33.202 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:33.203 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 10 librados: watch_flush enter 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 10 librados: watch_flush exit 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 10 client.23182.objecter close_session for osd.2 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7f8b90074460 msgr2=0x7f8b90076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6818/2342127620,v1:172.21.15.67:6819/2342127620] conn(0x7f8b90074460 0x7f8b90076900 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 20 client.23182.objecter put_session s=0x7f8b90073d90 osd=2 1 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 20 client.23182.objecter shutdown clearing up homeless session... 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 10 client.23182.objecter successfully canceled tick 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.202+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c870 msgr2=0x7f8b9006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c870 0x7f8b9006ed30 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8ba413c800 tx=0x7f8b98009040 comp rx=0 tx=0).stop 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 10 monclient: shutdown 2023-12-01T17:36:33.204 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ba40a8a20 msgr2=0x7f8ba4138f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 --2- 172.21.15.67:0/4099324449 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ba40a8a20 0x7f8ba4138f80 secure :-1 s=READY pgs=5891 cs=0 l=1 rev1=1 crypto rx=0x7f8b9c033000 tx=0x7f8b9c0046c0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8baa44d640 1 -- 172.21.15.67:0/4099324449 reap_dead start 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 shutdown_connections 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 >> 172.21.15.67:0/4099324449 conn(0x7f8ba40b7870 msgr2=0x7f8ba40b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 shutdown_connections 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 -- 172.21.15.67:0/4099324449 wait complete. 2023-12-01T17:36:33.205 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.203+0000 7f8babeb0640 1 librados: shutdown 2023-12-01T17:36:33.213 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-12-01T17:36:33.320 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.321+0000 7f43f1fd7640 10 monclient: get_monmap_and_config 2023-12-01T17:36:33.320 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.321+0000 7f43f1fd7640 10 monclient: build_initial_monmap 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.321+0000 7f43f1fd7640 10 monclient: monmap: 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:33.321902+0000 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:33.321902+0000 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:33.321 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:33.322 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:33.322 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.321+0000 7f43f1fd7640 1 Processor -- start 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.321+0000 7f43f1fd7640 1 -- start start 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.321+0000 7f43f1fd7640 10 monclient: init 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43ec0a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f43ec1461c0 0x7f43ec0a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f43ec1461c0 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:33.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43ec13f0f0 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec145620 con 0x7f43ec0a8a20 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43ec0aace0 con 0x7f43ec13f0f0 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43ec13fc60 con 0x7f43ec1461c0 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.322+0000 7f43f1fd7640 10 monclient(hunting): authenticate will time out at 3324.559953s 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.323+0000 7f43eb7fe640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f43ec0a8a20 0x7f43ec145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56698/0 (socket says 172.21.15.67:56698) 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.323+0000 7f43eb7fe640 1 -- 172.21.15.67:0/3417190125 learned_addr learned my addr 172.21.15.67:0/3417190125 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.323+0000 7f43eaffd640 1 --2- 172.21.15.67:0/3417190125 >> v2:172.21.15.92:3300/0 conn(0x7f43ec1461c0 0x7f43ec0a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.323+0000 7f43eaffd640 10 monclient(hunting): get_auth_request con 0x7f43ec1461c0 auth_method 0 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.323+0000 7f43eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.323+0000 7f43eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.323+0000 7f43eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.324+0000 7f43eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.325 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.324+0000 7f43eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.324+0000 7f43eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.325+0000 7f43ea7fc640 1 -- 172.21.15.67:0/3417190125 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143805516 0 0) 0x7f43ec145620 con 0x7f43ec0a8a20 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.325+0000 7f43ea7fc640 10 monclient(hunting): _negotiate 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.325+0000 7f43ea7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.325+0000 7f43ea7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.325+0000 7f43ea7fc640 10 monclient(hunting): my global_id is 15996 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.325+0000 7f43ea7fc640 1 -- 172.21.15.67:0/3417190125 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d4003520 con 0x7f43ec0a8a20 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 10 monclient(hunting): handle_auth_done global_id 23185 payload 274 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 1 -- 172.21.15.67:0/3417190125 >> v1:172.21.15.92:6789/0 conn(0x7f43ec13f0f0 legacy=0x7f43ec13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 1 -- 172.21.15.67:0/3417190125 >> v1:172.21.15.67:6789/0 conn(0x7f43ec0a8a20 legacy=0x7f43ec145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 10 monclient: _finish_hunting 0 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 1 monclient: found mon.noname-c 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.327 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.326+0000 7f43eaffd640 1 -- 172.21.15.67:0/3417190125 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43ec0aafc0 con 0x7f43ec1461c0 2023-12-01T17:36:33.328 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.329+0000 7f43eaffd640 1 --2- 172.21.15.67:0/3417190125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec0a8e00 secure :-1 s=READY pgs=5893 cs=0 l=1 rev1=1 crypto rx=0x7f43e00097a0 tx=0x7f43e002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fcaae1a07af5c6e server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 1 -- 172.21.15.67:0/3417190125 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0030610 con 0x7f43ec1461c0 2023-12-01T17:36:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: dump: 2023-12-01T17:36:33.333 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: _finish_auth 0 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: _check_auth_tickets 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 1 -- 172.21.15.67:0/3417190125 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e0030770 con 0x7f43ec1461c0 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:33.335 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 1 -- 172.21.15.67:0/3417190125 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e0030b00 con 0x7f43ec1461c0 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43ea7fc640 10 monclient: dump: 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.336 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.337 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43f1fd7640 5 monclient: authenticate success, global_id 23185 2023-12-01T17:36:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43f1fd7640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:33.337 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.333+0000 7f43f1fd7640 4 set_mon_vals no callback set 2023-12-01T17:36:33.340 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 10 monclient: shutdown 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 1 -- 172.21.15.67:0/3417190125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 msgr2=0x7f43ec0a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/3417190125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec0a8e00 secure :-1 s=READY pgs=5893 cs=0 l=1 rev1=1 crypto rx=0x7f43e00097a0 tx=0x7f43e002b960 comp rx=0 tx=0).stop 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 1 -- 172.21.15.67:0/3417190125 shutdown_connections 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/3417190125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec0a8e00 unknown :-1 s=CLOSED pgs=5893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 1 -- 172.21.15.67:0/3417190125 >> 172.21.15.67:0/3417190125 conn(0x7f43ec0b7890 msgr2=0x7f43ec0b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 1 -- 172.21.15.67:0/3417190125 shutdown_connections 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.334+0000 7f43f1fd7640 1 -- 172.21.15.67:0/3417190125 wait complete. 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 10 monclient: build_initial_monmap 2023-12-01T17:36:33.341 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 10 monclient: monmap: 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 1 librados: starting msgr at 2023-12-01T17:36:33.342 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 1 librados: starting objecter 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 1 Processor -- start 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 1 -- start start 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 1 librados: setting wanted keys 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 1 librados: calling monclient init 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.335+0000 7f43f1fd7640 10 monclient: init 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f43ec0a8a20 0x7f43ec142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f43ec0a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ec13f0f0 0x7f43ec143730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.343 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): picked mon.noname-a con 0x7f43ec13f0f0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): picked mon.noname-b con 0x7f43ec1461c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec1432a0 con 0x7f43ec13f0f0 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec141b80 con 0x7f43ec1461c0 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43ec141e60 con 0x7f43ec0a8a20 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43f1fd7640 10 monclient(hunting): authenticate will time out at 3324.573598s 2023-12-01T17:36:33.344 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43eb7fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f43ec0a8a20 0x7f43ec142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43eaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ec13f0f0 0x7f43ec143730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.336+0000 7f43ebfff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ec13f0f0 0x7f43ec143730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37846/0 (socket says 172.21.15.67:37846) 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 1 -- 172.21.15.67:0/242823744 learned_addr learned my addr 172.21.15.67:0/242823744 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 10 monclient(hunting): get_auth_request con 0x7f43ec13f0f0 auth_method 0 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eb7fe640 10 monclient(hunting): get_auth_request con 0x7f43ec0a8a20 auth_method 0 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eb7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.345 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eb7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43ebfff640 10 monclient(hunting): get_auth_request con 0x7f43ec1461c0 auth_method 0 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43ebfff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43ebfff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.337+0000 7f43eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 10 monclient(hunting): handle_auth_done global_id 23188 payload 824 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 1 -- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f43ec0a8a20 msgr2=0x7f43ec142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f43ec0a8a20 0x7f43ec142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 1 -- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ec13f0f0 msgr2=0x7f43ec143730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ec13f0f0 0x7f43ec143730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 10 monclient: _finish_hunting 0 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 1 monclient: found mon.noname-b 2023-12-01T17:36:33.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:33.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.348 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.340+0000 7f43ebfff640 1 -- 172.21.15.67:0/242823744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e0009400 con 0x7f43ec1461c0 2023-12-01T17:36:33.349 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.342+0000 7f43ebfff640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec143c70 secure :-1 s=READY pgs=5894 cs=0 l=1 rev1=1 crypto rx=0x7f43e400c690 tx=0x7f43e400cb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.349 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.342+0000 7f43d3fff640 10 client.?.objecter ms_handle_connect 0x7f43ec1461c0 2023-12-01T17:36:33.349 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.342+0000 7f43d3fff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 1 -- 172.21.15.67:0/242823744 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4003b70 con 0x7f43ec1461c0 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: dump: 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.350 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: _finish_auth 0 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 1 -- 172.21.15.67:0/242823744 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e4003cd0 con 0x7f43ec1461c0 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 1 -- 172.21.15.67:0/242823744 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e4005310 con 0x7f43ec1461c0 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 5 monclient: authenticate success, global_id 23188 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 10 monclient: _renew_subs 2023-12-01T17:36:33.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43ec142140 con 0x7f43ec1461c0 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43e9ffb640 4 set_mon_vals no callback set 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43d3fff640 10 monclient: dump: 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.352 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 10 client.23188.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 10 monclient: _renew_subs 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43ec141780 con 0x7f43ec1461c0 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.346+0000 7f43f1fd7640 1 librados: init done 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.347+0000 7f43f1fd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.347+0000 7f43f1fd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.347+0000 7f43f1fd7640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.347+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43ec143410 con 0x7f43ec1461c0 2023-12-01T17:36:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.586+0000 7f43d3fff640 1 -- 172.21.15.67:0/242823744 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f43e4005470 con 0x7f43ec1461c0 2023-12-01T17:36:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.586+0000 7f43d3fff640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f43d806c870 0x7f43d806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.586+0000 7f43eb7fe640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f43d806c870 0x7f43d806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.586 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.586+0000 7f43eb7fe640 10 monclient: get_auth_request con 0x7f43d806c870 auth_method 0 2023-12-01T17:36:33.587 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.587+0000 7f43d3fff640 10 client.23188.objecter ms_handle_connect 0x7f43d806c870 2023-12-01T17:36:33.587 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.587+0000 7f43eb7fe640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f43d806c870 0x7f43d806ed30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005880 tx=0x7f43dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.588+0000 7f43d3fff640 1 -- 172.21.15.67:0/242823744 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f43e4058b50 con 0x7f43ec1461c0 2023-12-01T17:36:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.588+0000 7f43d3fff640 10 client.23188.objecter ms_dispatch 0x7f43ec000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.588+0000 7f43d3fff640 3 client.23188.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.588+0000 7f43d3fff640 3 client.23188.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:33.588 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.588+0000 7f43d3fff640 20 client.23188.objecter dump_active .. 0 homeless 2023-12-01T17:36:33.597 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.598+0000 7f43d3fff640 1 -- 172.21.15.67:0/242823744 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f43e4054e90 con 0x7f43ec1461c0 2023-12-01T17:36:33.597 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.598+0000 7f43d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:33.597 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.598+0000 7f43d3fff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:33.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.781+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f43ec14d890 con 0x7f43d806c870 2023-12-01T17:36:33.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.782+0000 7f43d3fff640 1 -- 172.21.15.67:0/242823744 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f43dc00f820 con 0x7f43d806c870 2023-12-01T17:36:33.781 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 2 to deep-scrub 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 10 librados: watch_flush enter 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 10 librados: watch_flush exit 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 20 client.23188.objecter shutdown clearing up homeless session... 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 10 client.23188.objecter successfully canceled tick 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f43d806c870 msgr2=0x7f43d806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f43d806c870 0x7f43d806ed30 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f43dc005880 tx=0x7f43dc009040 comp rx=0 tx=0).stop 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 10 monclient: shutdown 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 msgr2=0x7f43ec143c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec143c70 secure :-1 s=READY pgs=5894 cs=0 l=1 rev1=1 crypto rx=0x7f43e400c690 tx=0x7f43e400cb50 comp rx=0 tx=0).stop 2023-12-01T17:36:33.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 shutdown_connections 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f43ec1461c0 0x7f43ec143c70 unknown :-1 s=CLOSED pgs=5894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43ec13f0f0 0x7f43ec143730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f43d806c870 0x7f43d806ed30 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 --2- 172.21.15.67:0/242823744 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f43ec0a8a20 0x7f43ec142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 >> 172.21.15.67:0/242823744 conn(0x7f43ec0b7890 msgr2=0x7f43ec145ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 shutdown_connections 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 -- 172.21.15.67:0/242823744 wait complete. 2023-12-01T17:36:33.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.784+0000 7f43f1fd7640 1 librados: shutdown 2023-12-01T17:36:33.795 INFO:tasks.ceph:Scrubbing osd.3 2023-12-01T17:36:33.795 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.910+0000 7f4a75815640 10 monclient: get_monmap_and_config 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.910+0000 7f4a75815640 10 monclient: build_initial_monmap 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.910+0000 7f4a75815640 10 monclient: monmap: 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:33.911286+0000 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:33.911286+0000 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.910 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:33.911 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:33.911 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:33.911 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:33.911 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:33.911 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 1 Processor -- start 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 1 -- start start 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient: init 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a700a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a701461c0 0x7f4a700a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient(hunting): picked mon.noname-b con 0x7f4a701461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:33.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a7013f0f0 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a70145620 con 0x7f4a7013f0f0 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a700aace0 con 0x7f4a700a8a20 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700aafc0 con 0x7f4a701461c0 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.911+0000 7f4a75815640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.912+0000 7f4a75815640 10 monclient(hunting): authenticate will time out at 3325.149410s 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.912+0000 7f4a6effd640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f4a700a8a20 0x7f4a70145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:36952/0 (socket says 172.21.15.67:36952) 2023-12-01T17:36:33.913 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.912+0000 7f4a6effd640 1 -- 172.21.15.67:0/3044308108 learned_addr learned my addr 172.21.15.67:0/3044308108 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.915+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405045198 0 0) 0x7f4a70145620 con 0x7f4a7013f0f0 2023-12-01T17:36:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.915+0000 7f4a6dffb640 10 monclient(hunting): _negotiate 2023-12-01T17:36:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.915+0000 7f4a6dffb640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.915+0000 7f4a6dffb640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.915+0000 7f4a6dffb640 10 monclient(hunting): my global_id is 16002 2023-12-01T17:36:33.915 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.915+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a58003520 con 0x7f4a7013f0f0 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4236141197 0 0) 0x7f4a58003520 con 0x7f4a7013f0f0 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a701461c0 msgr2=0x7f4a700a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a701461c0 0x7f4a700a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 >> v1:172.21.15.92:6789/0 conn(0x7f4a700a8a20 legacy=0x7f4a70145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: _finish_hunting 0 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 monclient: found mon.noname-a-legacy 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.67:6789/0 2023-12-01T17:36:33.920 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a700b2210 con 0x7f4a7013f0f0 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3910818764 0 0) 0x7f4a680019f0 con 0x7f4a7013f0f0 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: dump: 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.921 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] but i'm connected to v1:172.21.15.67:6789/0, reconnecting 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 >> v1:172.21.15.67:6789/0 conn(0x7f4a7013f0f0 legacy=0x7f4a7013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a58003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): picked mon.b con 0x7f4a580039a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a701461c0 0x7f4a580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): picked mon.c con 0x7f4a701461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a58004dd0 0x7f4a58005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): picked mon.a con 0x7f4a58004dd0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:33.922 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a700aafc0 con 0x7f4a58004dd0 2023-12-01T17:36:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5800bbe0 con 0x7f4a580039a0 2023-12-01T17:36:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5800bda0 con 0x7f4a701461c0 2023-12-01T17:36:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:33.923 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.920+0000 7f4a6dffb640 10 monclient(hunting): _finish_auth 0 2023-12-01T17:36:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.921+0000 7f4a6effd640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a58003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.921+0000 7f4a6effd640 10 monclient(hunting): get_auth_request con 0x7f4a580039a0 auth_method 0 2023-12-01T17:36:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.921+0000 7f4a6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.921+0000 7f4a6effd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.921+0000 7f4a6effd640 10 monclient(hunting): _init_auth already have auth, reseting 2023-12-01T17:36:33.924 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.922+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.922+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.922+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.923+0000 7f4a6f7fe640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a58004dd0 0x7f4a58005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 10 monclient(hunting): handle_auth_done global_id 16002 payload 293 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 1 -- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a701461c0 msgr2=0x7f4a580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a701461c0 0x7f4a580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 1 -- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a58004dd0 msgr2=0x7f4a58005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a58004dd0 0x7f4a58005250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 10 monclient: _finish_hunting 0 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 1 monclient: found mon.b 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 1 -- 172.21.15.67:0/3044308108 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58003520 con 0x7f4a580039a0 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 10 monclient: _finish_auth 0 2023-12-01T17:36:33.925 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a75815640 5 monclient: authenticate success, global_id 16002 2023-12-01T17:36:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.924+0000 7f4a75815640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:33.926 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.926+0000 7f4a6effd640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a58003dc0 secure :-1 s=READY pgs=5896 cs=0 l=1 rev1=1 crypto rx=0x7f4a60001810 tx=0x7f4a580057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b51a5c3cd6eab56 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a6002f070 con 0x7f4a580039a0 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 10 monclient: dump: 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.929 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a600048a0 con 0x7f4a580039a0 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 1 -- 172.21.15.67:0/3044308108 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60004c50 con 0x7f4a580039a0 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a75815640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a75815640 4 set_mon_vals no callback set 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.929+0000 7f4a6dffb640 10 monclient: dump: 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.930 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.931 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.931 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 10 monclient: shutdown 2023-12-01T17:36:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 1 -- 172.21.15.67:0/3044308108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 msgr2=0x7f4a58003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 1 --2- 172.21.15.67:0/3044308108 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a58003dc0 secure :-1 s=READY pgs=5896 cs=0 l=1 rev1=1 crypto rx=0x7f4a60001810 tx=0x7f4a580057f0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a6f7fe640 1 -- 172.21.15.67:0/3044308108 reap_dead start 2023-12-01T17:36:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 1 -- 172.21.15.67:0/3044308108 shutdown_connections 2023-12-01T17:36:33.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 1 -- 172.21.15.67:0/3044308108 >> 172.21.15.67:0/3044308108 conn(0x7f4a700b7890 msgr2=0x7f4a700b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 1 -- 172.21.15.67:0/3044308108 shutdown_connections 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.930+0000 7f4a75815640 1 -- 172.21.15.67:0/3044308108 wait complete. 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.931+0000 7f4a75815640 10 monclient: build_initial_monmap 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.931+0000 7f4a75815640 10 monclient: monmap: 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:33.935 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.931+0000 7f4a75815640 1 librados: starting msgr at 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.931+0000 7f4a75815640 1 librados: starting objecter 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 1 Processor -- start 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 1 -- start start 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 1 librados: setting wanted keys 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 1 librados: calling monclient init 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 10 monclient: init 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a701da980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 10 monclient(hunting): picked mon.noname-b con 0x7f4a580039a0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a701db3a0 0x7f4a70143500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 10 monclient(hunting): picked mon.noname-a con 0x7f4a701db3a0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:33.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.932+0000 7f4a75815640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a70142a30 0x7f4a70143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 10 monclient(hunting): picked mon.noname-c con 0x7f4a70142a30 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70140110 con 0x7f4a701db3a0 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70144db0 con 0x7f4a580039a0 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a70142fe0 con 0x7f4a70142a30 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a75815640 10 monclient(hunting): authenticate will time out at 3325.170511s 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6effd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a701da980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6e7fc640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a701db3a0 0x7f4a70143500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6effd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a701da980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42654/0 (socket says 172.21.15.67:42654) 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6effd640 1 -- 172.21.15.67:0/489234370 learned_addr learned my addr 172.21.15.67:0/489234370 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:33.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6effd640 10 monclient(hunting): get_auth_request con 0x7f4a580039a0 auth_method 0 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6effd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.933+0000 7f4a6effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6e7fc640 10 monclient(hunting): get_auth_request con 0x7f4a701db3a0 auth_method 0 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6e7fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6e7fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.934+0000 7f4a6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.936+0000 7f4a6f7fe640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a70142a30 0x7f4a70143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.936+0000 7f4a6f7fe640 10 monclient(hunting): get_auth_request con 0x7f4a70142a30 auth_method 0 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.936+0000 7f4a6f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.936+0000 7f4a6f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.936+0000 7f4a6f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.937+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.937+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 10 monclient(hunting): handle_auth_done global_id 23191 payload 824 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 1 -- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a70142a30 msgr2=0x7f4a70143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f4a70142a30 0x7f4a70143a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 1 -- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a701db3a0 msgr2=0x7f4a70143500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a701db3a0 0x7f4a70143500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 10 monclient: _finish_hunting 0 2023-12-01T17:36:33.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 1 monclient: found mon.noname-b 2023-12-01T17:36:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.938+0000 7f4a6effd640 1 -- 172.21.15.67:0/489234370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a70143f80 con 0x7f4a580039a0 2023-12-01T17:36:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.941+0000 7f4a6effd640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a701da980 secure :-1 s=READY pgs=5897 cs=0 l=1 rev1=1 crypto rx=0x7f4a60001810 tx=0x7f4a60001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.941+0000 7f4a57fff640 10 client.?.objecter ms_handle_connect 0x7f4a580039a0 2023-12-01T17:36:33.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.941+0000 7f4a57fff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60000ed0 con 0x7f4a580039a0 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: dump: 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.945 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: _finish_auth 0 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a60035470 con 0x7f4a580039a0 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 5 monclient: authenticate success, global_id 23191 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 10 monclient: _renew_subs 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a700b0780 con 0x7f4a580039a0 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a60035800 con 0x7f4a580039a0 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:33.946 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a57fff640 10 monclient: dump: 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a6d7fa640 4 set_mon_vals no callback set 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 10 client.23191.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 10 monclient: _renew_subs 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.947 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a700b0c30 con 0x7f4a580039a0 2023-12-01T17:36:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.945+0000 7f4a75815640 1 librados: init done 2023-12-01T17:36:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.946+0000 7f4a75815640 10 client.23191.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.946+0000 7f4a75815640 15 client.23191.objecter _session_command_op_assign -1 1 2023-12-01T17:36:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.946+0000 7f4a75815640 20 client.23191.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.946+0000 7f4a75815640 10 client.23191.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.946+0000 7f4a75815640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:33.948 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.946+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4a701431a0 con 0x7f4a580039a0 2023-12-01T17:36:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.964+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4a60035960 con 0x7f4a580039a0 2023-12-01T17:36:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.965+0000 7f4a57fff640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f4a5c06c820 0x7f4a5c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.965+0000 7f4a6e7fc640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f4a5c06c820 0x7f4a5c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.965+0000 7f4a6e7fc640 10 monclient: get_auth_request con 0x7f4a5c06c820 auth_method 0 2023-12-01T17:36:33.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.966+0000 7f4a6e7fc640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f4a5c06c820 0x7f4a5c06ece0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4a701419b0 tx=0x7f4a64006f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.966+0000 7f4a57fff640 10 client.23191.objecter ms_handle_connect 0x7f4a5c06c820 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f4a60079df0 con 0x7f4a580039a0 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 10 client.23191.objecter ms_dispatch 0x7f4a70000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 3 client.23191.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 3 client.23191.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 10 client.23191.objecter checking command 1 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f4a5c074380 0x7f4a5c076820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 20 client.23191.objecter _get_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 20 client.23191.objecter put_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 15 client.23191.objecter _session_command_op_remove -1 1 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 20 client.23191.objecter _get_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:33.968 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 20 client.23191.objecter get_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 15 client.23191.objecter _session_command_op_assign 3 1 2023-12-01T17:36:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 20 client.23191.objecter put_session s=0x7f4a5c073cb0 osd=3 4 2023-12-01T17:36:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 10 client.23191.objecter _send_command 1 2023-12-01T17:36:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 --> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4a5c076ef0 con 0x7f4a5c074380 2023-12-01T17:36:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.968+0000 7f4a57fff640 20 client.23191.objecter dump_active .. 0 homeless 2023-12-01T17:36:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.969+0000 7f4a6f7fe640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f4a5c074380 0x7f4a5c076820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:33.969 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.969+0000 7f4a6f7fe640 10 monclient: get_auth_request con 0x7f4a5c074380 auth_method 0 2023-12-01T17:36:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.973+0000 7f4a57fff640 10 client.23191.objecter ms_handle_connect 0x7f4a5c074380 2023-12-01T17:36:33.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:33.973+0000 7f4a6f7fe640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f4a5c074380 0x7f4a5c076820 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.000+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 <== osd.3 v2:172.21.15.67:6802/1625009733 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f4a5c076ef0 con 0x7f4a5c074380 2023-12-01T17:36:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.000+0000 7f4a57fff640 10 client.23191.objecter ms_dispatch 0x7f4a70000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.000+0000 7f4a57fff640 10 client.23191.objecter _finish_command 1 = system:0 2023-12-01T17:36:33.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.000+0000 7f4a57fff640 20 client.23191.objecter put_session s=0x7f4a5c073cb0 osd=3 4 2023-12-01T17:36:34.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.000+0000 7f4a57fff640 15 client.23191.objecter _session_command_op_remove 3 1 2023-12-01T17:36:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 10 client.23191.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 15 client.23191.objecter _session_command_op_assign -1 2 2023-12-01T17:36:34.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 20 client.23191.objecter _get_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 20 client.23191.objecter put_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 20 client.23191.objecter _get_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 15 client.23191.objecter _session_command_op_remove -1 2 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 20 client.23191.objecter get_session s=0x7f4a5c073cb0 osd=3 3 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 15 client.23191.objecter _session_command_op_assign 3 2 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 20 client.23191.objecter put_session s=0x7f4a5c073cb0 osd=3 4 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 10 client.23191.objecter _send_command 2 2023-12-01T17:36:34.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.014+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 --> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4a7004e840 con 0x7f4a5c074380 2023-12-01T17:36:34.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a57fff640 1 -- 172.21.15.67:0/489234370 <== osd.3 v2:172.21.15.67:6802/1625009733 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4a7004e840 con 0x7f4a5c074380 2023-12-01T17:36:34.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a57fff640 10 client.23191.objecter ms_dispatch 0x7f4a70000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:34.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a57fff640 10 client.23191.objecter _finish_command 2 = system:0 2023-12-01T17:36:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a57fff640 20 client.23191.objecter put_session s=0x7f4a5c073cb0 osd=3 4 2023-12-01T17:36:34.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a57fff640 15 client.23191.objecter _session_command_op_remove 3 2 2023-12-01T17:36:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:34.029 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:34.029 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 10 librados: watch_flush enter 2023-12-01T17:36:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 10 librados: watch_flush exit 2023-12-01T17:36:34.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 10 client.23191.objecter close_session for osd.3 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f4a5c074380 msgr2=0x7f4a5c076820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6802/1625009733,v1:172.21.15.67:6803/1625009733] conn(0x7f4a5c074380 0x7f4a5c076820 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 20 client.23191.objecter put_session s=0x7f4a5c073cb0 osd=3 1 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 20 client.23191.objecter shutdown clearing up homeless session... 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 10 client.23191.objecter successfully canceled tick 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f4a5c06c820 msgr2=0x7f4a5c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f4a5c06c820 0x7f4a5c06ece0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f4a701419b0 tx=0x7f4a64006f90 comp rx=0 tx=0).stop 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 10 monclient: shutdown 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 20 monclient: shutdown canceling and discarding version request 1 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 msgr2=0x7f4a701da980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 --2- 172.21.15.67:0/489234370 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f4a580039a0 0x7f4a701da980 secure :-1 s=READY pgs=5897 cs=0 l=1 rev1=1 crypto rx=0x7f4a60001810 tx=0x7f4a60001120 comp rx=0 tx=0).stop 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a6f7fe640 1 -- 172.21.15.67:0/489234370 reap_dead start 2023-12-01T17:36:34.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 shutdown_connections 2023-12-01T17:36:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 >> 172.21.15.67:0/489234370 conn(0x7f4a700b7890 msgr2=0x7f4a700b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 shutdown_connections 2023-12-01T17:36:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 -- 172.21.15.67:0/489234370 wait complete. 2023-12-01T17:36:34.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.029+0000 7f4a75815640 1 librados: shutdown 2023-12-01T17:36:34.043 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-12-01T17:36:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.152+0000 7fb4361c5640 10 monclient: get_monmap_and_config 2023-12-01T17:36:34.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.152+0000 7fb4361c5640 10 monclient: build_initial_monmap 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.152+0000 7fb4361c5640 10 monclient: monmap: 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:34.153057+0000 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:34.153057+0000 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:34.152 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 Processor -- start 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 -- start start 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient: init 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4300b0b80 0x7fb430071990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): picked mon.noname-a con 0x7fb4300b0b80 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb43013d0c0 0x7fb43013d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): picked mon.noname-c con 0x7fb43013d0c0 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb430143240 0x7fb4301436d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.155 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): picked mon.noname-b con 0x7fb430143240 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4300b0f60 con 0x7fb4300b0b80 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb430072540 con 0x7fb43013d0c0 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4300a8b70 con 0x7fb430143240 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.153+0000 7fb4361c5640 10 monclient(hunting): authenticate will time out at 3325.391173s 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb426ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb43013d0c0 0x7fb43013d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42f7fe640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fb4300b0b80 0x7fb430071990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb430143240 0x7fb4301436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb426ffd640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fb43013d0c0 0x7fb43013d500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42660/0 (socket says 172.21.15.67:42660) 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb426ffd640 1 -- 172.21.15.67:0/1851451923 learned_addr learned my addr 172.21.15.67:0/1851451923 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb426ffd640 10 monclient(hunting): get_auth_request con 0x7fb43013d0c0 auth_method 0 2023-12-01T17:36:34.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb426ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb426ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42ffff640 10 monclient(hunting): get_auth_request con 0x7fb430143240 auth_method 0 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42ffff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42f7fe640 10 monclient(hunting): get_auth_request con 0x7fb4300b0b80 auth_method 0 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.154+0000 7fb42f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb426ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb426ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb426ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb42ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb42ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb42ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb42f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb42f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.155+0000 7fb42f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 10 monclient(hunting): handle_auth_done global_id 17942 payload 274 2023-12-01T17:36:34.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 1 -- 172.21.15.67:0/1851451923 >> v2:172.21.15.92:3300/0 conn(0x7fb43013d0c0 msgr2=0x7fb43013d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 1 --2- 172.21.15.67:0/1851451923 >> v2:172.21.15.92:3300/0 conn(0x7fb43013d0c0 0x7fb43013d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 1 -- 172.21.15.67:0/1851451923 >> v2:172.21.15.67:3300/0 conn(0x7fb4300b0b80 msgr2=0x7fb430071990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 1 --2- 172.21.15.67:0/1851451923 >> v2:172.21.15.67:3300/0 conn(0x7fb4300b0b80 0x7fb430071990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 10 monclient: _finish_hunting 0 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 1 monclient: found mon.noname-b 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42ffff640 1 -- 172.21.15.67:0/1851451923 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4300a8dc0 con 0x7fb430143240 2023-12-01T17:36:34.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb426ffd640 1 --2- 172.21.15.67:0/1851451923 >> v2:172.21.15.92:3300/0 conn(0x7fb43013d0c0 0x7fb43013d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-01T17:36:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.157+0000 7fb42f7fe640 1 --2- 172.21.15.67:0/1851451923 >> v2:172.21.15.67:3300/0 conn(0x7fb4300b0b80 0x7fb430071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-12-01T17:36:34.160 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.158+0000 7fb42ffff640 1 --2- 172.21.15.67:0/1851451923 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb430143240 0x7fb4301436d0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fb43006cf20 tx=0x7fb42802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cbf8a1eeb0bf3c27 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 1 -- 172.21.15.67:0/1851451923 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42802cbc0 con 0x7fb430143240 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: dump: 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.162 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: _finish_auth 0 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 1 -- 172.21.15.67:0/1851451923 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42802cd20 con 0x7fb430143240 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb4361c5640 5 monclient: authenticate success, global_id 17942 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb4361c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb4361c5640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 1 -- 172.21.15.67:0/1851451923 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428034660 con 0x7fb430143240 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb4361c5640 4 set_mon_vals no callback set 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.163 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.162+0000 7fb42effd640 10 monclient: dump: 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.164 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 10 monclient: shutdown 2023-12-01T17:36:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 -- 172.21.15.67:0/1851451923 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb430143240 msgr2=0x7fb4301436d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.167 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 --2- 172.21.15.67:0/1851451923 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb430143240 0x7fb4301436d0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fb43006cf20 tx=0x7fb42802be10 comp rx=0 tx=0).stop 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 -- 172.21.15.67:0/1851451923 shutdown_connections 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 --2- 172.21.15.67:0/1851451923 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb430143240 0x7fb4301436d0 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 --2- 172.21.15.67:0/1851451923 >> v2:172.21.15.92:3300/0 conn(0x7fb43013d0c0 0x7fb43013d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 --2- 172.21.15.67:0/1851451923 >> v2:172.21.15.67:3300/0 conn(0x7fb4300b0b80 0x7fb430071990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 -- 172.21.15.67:0/1851451923 >> 172.21.15.67:0/1851451923 conn(0x7fb4300b3ef0 msgr2=0x7fb4300b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 -- 172.21.15.67:0/1851451923 shutdown_connections 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.163+0000 7fb4361c5640 1 -- 172.21.15.67:0/1851451923 wait complete. 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 10 monclient: build_initial_monmap 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 10 monclient: monmap: 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:34.168 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 1 librados: starting msgr at 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 1 librados: starting objecter 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 1 Processor -- start 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 1 -- start start 2023-12-01T17:36:34.169 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 1 librados: setting wanted keys 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 1 librados: calling monclient init 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.164+0000 7fb4361c5640 10 monclient: init 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4300b0b80 0x7fb43006f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): picked mon.noname-b con 0x7fb4300b0b80 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb43013d0c0 0x7fb43006fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): picked mon.noname-c con 0x7fb43013d0c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb430143240 0x7fb4301d75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): picked mon.noname-a con 0x7fb430143240 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb430070240 con 0x7fb430143240 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.170 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb430136580 con 0x7fb4300b0b80 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4301366f0 con 0x7fb43013d0c0 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb4361c5640 10 monclient(hunting): authenticate will time out at 3325.402728s 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb426ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb43013d0c0 0x7fb43006fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.165+0000 7fb42ffff640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb430143240 0x7fb4301d75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4300b0b80 0x7fb43006f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb43013d0c0 0x7fb43006fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:45792/0 (socket says 172.21.15.67:45792) 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 1 -- 172.21.15.67:0/2638300367 learned_addr learned my addr 172.21.15.67:0/2638300367 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 10 monclient(hunting): get_auth_request con 0x7fb43013d0c0 auth_method 0 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 10 monclient(hunting): get_auth_request con 0x7fb4300b0b80 auth_method 0 2023-12-01T17:36:34.171 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42ffff640 10 monclient(hunting): get_auth_request con 0x7fb430143240 auth_method 0 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42ffff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb42f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.166+0000 7fb426ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.167+0000 7fb426ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.167+0000 7fb42ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.167+0000 7fb42ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.172 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.167+0000 7fb42ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 10 monclient(hunting): handle_auth_done global_id 23197 payload 824 2023-12-01T17:36:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 1 -- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb43013d0c0 msgr2=0x7fb43006fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb43013d0c0 0x7fb43006fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 1 -- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb430143240 msgr2=0x7fb4301d75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb430143240 0x7fb4301d75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.173 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 10 monclient: _finish_hunting 0 2023-12-01T17:36:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 1 monclient: found mon.noname-b 2023-12-01T17:36:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.169+0000 7fb42f7fe640 1 -- 172.21.15.67:0/2638300367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb420009400 con 0x7fb4300b0b80 2023-12-01T17:36:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.171+0000 7fb42f7fe640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4300b0b80 0x7fb43006f2b0 secure :-1 s=READY pgs=5898 cs=0 l=1 rev1=1 crypto rx=0x7fb42000e930 tx=0x7fb42000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.171+0000 7fb42cff9640 10 client.?.objecter ms_handle_connect 0x7fb4300b0b80 2023-12-01T17:36:34.174 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.171+0000 7fb42cff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 1 -- 172.21.15.67:0/2638300367 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb42000ca00 con 0x7fb4300b0b80 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: dump: 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.175 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: _finish_auth 0 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: _check_auth_tickets 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 1 -- 172.21.15.67:0/2638300367 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42000cb60 con 0x7fb4300b0b80 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 5 monclient: authenticate success, global_id 23197 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 10 monclient: _renew_subs 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4180094d0 con 0x7fb4300b0b80 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:34.176 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 1 -- 172.21.15.67:0/2638300367 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420013470 con 0x7fb4300b0b80 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42cff9640 10 monclient: dump: 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb42e7fc640 4 set_mon_vals no callback set 2023-12-01T17:36:34.177 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 10 client.23197.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 10 monclient: _renew_subs 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb428009a70 con 0x7fb4300b0b80 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.175+0000 7fb4361c5640 1 librados: init done 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.176+0000 7fb4361c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.176+0000 7fb4361c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.176+0000 7fb4361c5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.178 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.176+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43006f840 con 0x7fb4300b0b80 2023-12-01T17:36:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.419+0000 7fb42cff9640 1 -- 172.21.15.67:0/2638300367 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fb4200135d0 con 0x7fb4300b0b80 2023-12-01T17:36:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.419+0000 7fb42cff9640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fb41006c820 0x7fb41006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.419+0000 7fb426ffd640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fb41006c820 0x7fb41006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.419+0000 7fb426ffd640 10 monclient: get_auth_request con 0x7fb41006c820 auth_method 0 2023-12-01T17:36:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.420+0000 7fb42cff9640 10 client.23197.objecter ms_handle_connect 0x7fb41006c820 2023-12-01T17:36:34.419 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.420+0000 7fb426ffd640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fb41006c820 0x7fb41006ece0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb4301d7500 tx=0x7fb418009240 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.423+0000 7fb42cff9640 1 -- 172.21.15.67:0/2638300367 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fb420058dd0 con 0x7fb4300b0b80 2023-12-01T17:36:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.423+0000 7fb42cff9640 10 client.23197.objecter ms_dispatch 0x7fb430000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.423+0000 7fb42cff9640 3 client.23197.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.423+0000 7fb42cff9640 3 client.23197.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:34.423 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.423+0000 7fb42cff9640 20 client.23197.objecter dump_active .. 0 homeless 2023-12-01T17:36:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.433+0000 7fb42cff9640 1 -- 172.21.15.67:0/2638300367 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb420055110 con 0x7fb4300b0b80 2023-12-01T17:36:34.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.433+0000 7fb42cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:34.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.433+0000 7fb42cff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:34.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.617+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb430072080 con 0x7fb41006c820 2023-12-01T17:36:34.617 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.618+0000 7fb42cff9640 1 -- 172.21.15.67:0/2638300367 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb430072080 con 0x7fb41006c820 2023-12-01T17:36:34.617 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 3 to deep-scrub 2023-12-01T17:36:34.619 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.619+0000 7fb4361c5640 10 librados: watch_flush enter 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.619+0000 7fb4361c5640 10 librados: watch_flush exit 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.619+0000 7fb4361c5640 20 client.23197.objecter shutdown clearing up homeless session... 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 10 client.23197.objecter successfully canceled tick 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fb41006c820 msgr2=0x7fb41006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fb41006c820 0x7fb41006ece0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb4301d7500 tx=0x7fb418009240 comp rx=0 tx=0).stop 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 10 monclient: shutdown 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4300b0b80 msgr2=0x7fb43006f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4300b0b80 0x7fb43006f2b0 secure :-1 s=READY pgs=5898 cs=0 l=1 rev1=1 crypto rx=0x7fb42000e930 tx=0x7fb42000edf0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 shutdown_connections 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb430143240 0x7fb4301d75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fb41006c820 0x7fb41006ece0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.620 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fb43013d0c0 0x7fb43006fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.621 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 --2- 172.21.15.67:0/2638300367 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb4300b0b80 0x7fb43006f2b0 unknown :-1 s=CLOSED pgs=5898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.621 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 >> 172.21.15.67:0/2638300367 conn(0x7fb4300b3ef0 msgr2=0x7fb4301379f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:34.621 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 shutdown_connections 2023-12-01T17:36:34.621 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 -- 172.21.15.67:0/2638300367 wait complete. 2023-12-01T17:36:34.621 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.620+0000 7fb4361c5640 1 librados: shutdown 2023-12-01T17:36:34.631 INFO:tasks.ceph:Scrubbing osd.4 2023-12-01T17:36:34.631 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:34.741 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.742+0000 7f5a5695a640 10 monclient: get_monmap_and_config 2023-12-01T17:36:34.741 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.742+0000 7f5a5695a640 10 monclient: build_initial_monmap 2023-12-01T17:36:34.741 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.742+0000 7f5a5695a640 10 monclient: monmap: 2023-12-01T17:36:34.741 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:34.741 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.741 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:34.742796+0000 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:34.742796+0000 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:34.742 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.743 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.742+0000 7f5a5695a640 1 Processor -- start 2023-12-01T17:36:34.743 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.742+0000 7f5a5695a640 1 -- start start 2023-12-01T17:36:34.743 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.742+0000 7f5a5695a640 10 monclient: init 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a500a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a500a8e00 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a5013f5c0 0x7f5a50142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient(hunting): picked mon.noname-a con 0x7f5a5013f5c0 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a500b21f0 con 0x7f5a500a8a20 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a500a9ed0 con 0x7f5a500a8e00 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a500a9cc0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.743+0000 7f5a5695a640 10 monclient(hunting): authenticate will time out at 3325.980831s 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.744+0000 7f5a54ef7640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5a5013f5c0 0x7f5a50142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.744+0000 7f5a4f7fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f5a500a8e00 0x7f5a501423b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:36956/0 (socket says 172.21.15.67:36956) 2023-12-01T17:36:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.744+0000 7f5a4f7fe640 1 -- 172.21.15.67:0/993959029 learned_addr learned my addr 172.21.15.67:0/993959029 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.744+0000 7f5a54ef7640 10 monclient(hunting): get_auth_request con 0x7f5a5013f5c0 auth_method 0 2023-12-01T17:36:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.744+0000 7f5a54ef7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.744+0000 7f5a54ef7640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.744+0000 7f5a54ef7640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.745+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.745+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.745+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.746+0000 7f5a4effd640 1 -- 172.21.15.67:0/993959029 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771124251 0 0) 0x7f5a500a9ed0 con 0x7f5a500a8e00 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.746+0000 7f5a4effd640 10 monclient(hunting): _negotiate 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.746+0000 7f5a4effd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.746+0000 7f5a4effd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.746+0000 7f5a4effd640 10 monclient(hunting): my global_id is 23200 2023-12-01T17:36:34.746 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.746+0000 7f5a4effd640 1 -- 172.21.15.67:0/993959029 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a38003520 con 0x7f5a500a8e00 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_done global_id 16014 payload 274 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 1 -- 172.21.15.67:0/993959029 >> v1:172.21.15.92:6789/0 conn(0x7f5a500a8e00 legacy=0x7f5a501423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 1 -- 172.21.15.67:0/993959029 >> v1:172.21.15.67:6789/0 conn(0x7f5a500a8a20 legacy=0x7f5a5013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 10 monclient: _finish_hunting 0 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 1 monclient: found mon.noname-a 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:36:34.748 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.748+0000 7f5a54ef7640 1 -- 172.21.15.67:0/993959029 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a500aac70 con 0x7f5a5013f5c0 2023-12-01T17:36:34.751 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.751+0000 7f5a54ef7640 1 --2- 172.21.15.67:0/993959029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a5013f5c0 0x7f5a50142ac0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009d90 tx=0x7f5a4802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6752200ce425deb server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.755+0000 7f5a4effd640 1 -- 172.21.15.67:0/993959029 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4802cbd0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.755+0000 7f5a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:36:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: dump: 2023-12-01T17:36:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.755 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: _finish_auth 0 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 1 -- 172.21.15.67:0/993959029 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4802cd30 con 0x7f5a5013f5c0 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 5 monclient: authenticate success, global_id 16014 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:34.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 4 set_mon_vals no callback set 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 1 -- 172.21.15.67:0/993959029 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4802cea0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a4effd640 10 monclient: dump: 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.757 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.758 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 10 monclient: shutdown 2023-12-01T17:36:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:34.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 1 -- 172.21.15.67:0/993959029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a5013f5c0 msgr2=0x7f5a50142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 1 --2- 172.21.15.67:0/993959029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a5013f5c0 0x7f5a50142ac0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009d90 tx=0x7f5a4802be20 comp rx=0 tx=0).stop 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 1 -- 172.21.15.67:0/993959029 shutdown_connections 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 1 --2- 172.21.15.67:0/993959029 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a5013f5c0 0x7f5a50142ac0 unknown :-1 s=CLOSED pgs=3172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 1 -- 172.21.15.67:0/993959029 >> 172.21.15.67:0/993959029 conn(0x7f5a500b7870 msgr2=0x7f5a500b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 1 -- 172.21.15.67:0/993959029 shutdown_connections 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.756+0000 7f5a5695a640 1 -- 172.21.15.67:0/993959029 wait complete. 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 10 monclient: build_initial_monmap 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 10 monclient: monmap: 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:34.762 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 1 librados: starting msgr at 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 1 librados: starting objecter 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 1 Processor -- start 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 1 -- start start 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 1 librados: setting wanted keys 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 1 librados: calling monclient init 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.757+0000 7f5a5695a640 10 monclient: init 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a500a8a20 0x7f5a501399d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.763 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): picked mon.noname-c con 0x7f5a500a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a500a8e00 0x7f5a5013a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): picked mon.noname-a con 0x7f5a500a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a5013f5c0 0x7f5a5013a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): picked mon.noname-b con 0x7f5a5013f5c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a5013b590 con 0x7f5a500a8e00 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a500ab5c0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a500ab8a0 con 0x7f5a500a8a20 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a5695a640 10 monclient(hunting): authenticate will time out at 3325.995903s 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.758+0000 7f5a4f7fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a500a8e00 0x7f5a5013a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a5013f5c0 0x7f5a5013a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4ffff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a500a8a20 0x7f5a501399d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a5013f5c0 0x7f5a5013a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42680/0 (socket says 172.21.15.67:42680) 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 1 -- 172.21.15.67:0/3437266736 learned_addr learned my addr 172.21.15.67:0/3437266736 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 10 monclient(hunting): get_auth_request con 0x7f5a5013f5c0 auth_method 0 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4f7fe640 10 monclient(hunting): get_auth_request con 0x7f5a500a8e00 auth_method 0 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4f7fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4f7fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4ffff640 10 monclient(hunting): get_auth_request con 0x7f5a500a8a20 auth_method 0 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4ffff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4ffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.759+0000 7f5a4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.760+0000 7f5a4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.760+0000 7f5a4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.760+0000 7f5a4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 10 monclient(hunting): handle_auth_done global_id 23203 payload 824 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 1 -- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a500a8a20 msgr2=0x7f5a501399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f5a500a8a20 0x7f5a501399d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 1 -- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a500a8e00 msgr2=0x7f5a5013a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5a500a8e00 0x7f5a5013a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 10 monclient: _finish_hunting 0 2023-12-01T17:36:34.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 1 monclient: found mon.noname-b 2023-12-01T17:36:34.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:34.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.767 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.762+0000 7f5a54ef7640 1 -- 172.21.15.67:0/3437266736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a48009a70 con 0x7f5a5013f5c0 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.764+0000 7f5a54ef7640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a5013f5c0 0x7f5a5013a880 secure :-1 s=READY pgs=5900 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009eb0 tx=0x7f5a48004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.764+0000 7f5a4cff9640 10 client.?.objecter ms_handle_connect 0x7f5a5013f5c0 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.764+0000 7f5a4cff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4802cea0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: dump: 2023-12-01T17:36:34.768 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: _finish_auth 0 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: _check_auth_tickets 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4803e610 con 0x7f5a5013f5c0 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4803e9c0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.769 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4cff9640 10 monclient: dump: 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 5 monclient: authenticate success, global_id 23203 2023-12-01T17:36:34.770 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 monclient: _renew_subs 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a4e7fc640 4 set_mon_vals no callback set 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a5013adc0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 client.23203.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 monclient: _renew_subs 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a500ac180 con 0x7f5a5013f5c0 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 1 librados: init done 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 client.23203.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 15 client.23203.objecter _session_command_op_assign -1 1 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 20 client.23203.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 client.23203.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:34.771 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.768+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5a5013fb50 con 0x7f5a5013f5c0 2023-12-01T17:36:34.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.786+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f5a4803b050 con 0x7f5a5013f5c0 2023-12-01T17:36:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.786+0000 7f5a4cff9640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5a3c06c820 0x7f5a3c06ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.786+0000 7f5a4ffff640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5a3c06c820 0x7f5a3c06ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.786+0000 7f5a4ffff640 10 monclient: get_auth_request con 0x7f5a3c06c820 auth_method 0 2023-12-01T17:36:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.787+0000 7f5a4ffff640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5a3c06c820 0x7f5a3c06ece0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5a44001780 tx=0x7f5a44009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.787+0000 7f5a4cff9640 10 client.23203.objecter ms_handle_connect 0x7f5a3c06c820 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f5a4807aec0 con 0x7f5a5013f5c0 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 10 client.23203.objecter ms_dispatch 0x7f5a50000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 3 client.23203.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 3 client.23203.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 10 client.23203.objecter checking command 1 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f5a3c074410 0x7f5a3c0768b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 20 client.23203.objecter _get_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 20 client.23203.objecter put_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 15 client.23203.objecter _session_command_op_remove -1 1 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 20 client.23203.objecter _get_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 20 client.23203.objecter get_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 15 client.23203.objecter _session_command_op_assign 4 1 2023-12-01T17:36:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 20 client.23203.objecter put_session s=0x7f5a3c073d40 osd=4 4 2023-12-01T17:36:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 10 client.23203.objecter _send_command 1 2023-12-01T17:36:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 --> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a3c076f80 con 0x7f5a3c074410 2023-12-01T17:36:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4cff9640 20 client.23203.objecter dump_active .. 0 homeless 2023-12-01T17:36:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.789+0000 7f5a4f7fe640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f5a3c074410 0x7f5a3c0768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.790+0000 7f5a4f7fe640 10 monclient: get_auth_request con 0x7f5a3c074410 auth_method 0 2023-12-01T17:36:34.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.790+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=1099) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a48077200 con 0x7f5a5013f5c0 2023-12-01T17:36:34.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.790+0000 7f5a4cff9640 10 monclient: handle_get_version_reply finishing 1 version 1099 2023-12-01T17:36:34.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.792+0000 7f5a4f7fe640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f5a3c074410 0x7f5a3c0768b0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.792 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.793+0000 7f5a4cff9640 10 client.23203.objecter ms_handle_connect 0x7f5a3c074410 2023-12-01T17:36:34.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.816+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== osd.4 v2:172.21.15.92:6816/2070368587 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5a3c076f80 con 0x7f5a3c074410 2023-12-01T17:36:34.815 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.816+0000 7f5a4cff9640 10 client.23203.objecter ms_dispatch 0x7f5a50000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:34.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.816+0000 7f5a4cff9640 10 client.23203.objecter _finish_command 1 = system:0 2023-12-01T17:36:34.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.816+0000 7f5a4cff9640 20 client.23203.objecter put_session s=0x7f5a3c073d40 osd=4 4 2023-12-01T17:36:34.816 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.816+0000 7f5a4cff9640 15 client.23203.objecter _session_command_op_remove 4 1 2023-12-01T17:36:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 10 client.23203.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 15 client.23203.objecter _session_command_op_assign -1 2 2023-12-01T17:36:34.828 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 20 client.23203.objecter _get_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 20 client.23203.objecter put_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 20 client.23203.objecter _get_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 15 client.23203.objecter _session_command_op_remove -1 2 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 20 client.23203.objecter get_session s=0x7f5a3c073d40 osd=4 3 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 15 client.23203.objecter _session_command_op_assign 4 2 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 20 client.23203.objecter put_session s=0x7f5a3c073d40 osd=4 4 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 10 client.23203.objecter _send_command 2 2023-12-01T17:36:34.829 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.829+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 --> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5a500b3f40 con 0x7f5a3c074410 2023-12-01T17:36:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a4cff9640 1 -- 172.21.15.67:0/3437266736 <== osd.4 v2:172.21.15.92:6816/2070368587 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5a500b3f40 con 0x7f5a3c074410 2023-12-01T17:36:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a4cff9640 10 client.23203.objecter ms_dispatch 0x7f5a50000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a4cff9640 10 client.23203.objecter _finish_command 2 = system:0 2023-12-01T17:36:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a4cff9640 20 client.23203.objecter put_session s=0x7f5a3c073d40 osd=4 4 2023-12-01T17:36:34.841 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a4cff9640 15 client.23203.objecter _session_command_op_remove 4 2 2023-12-01T17:36:34.842 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:34.842 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:34.842 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 10 librados: watch_flush enter 2023-12-01T17:36:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 10 librados: watch_flush exit 2023-12-01T17:36:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 10 client.23203.objecter close_session for osd.4 2023-12-01T17:36:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f5a3c074410 msgr2=0x7f5a3c0768b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:6816/2070368587,v1:172.21.15.92:6817/2070368587] conn(0x7f5a3c074410 0x7f5a3c0768b0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 20 client.23203.objecter put_session s=0x7f5a3c073d40 osd=4 1 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 20 client.23203.objecter shutdown clearing up homeless session... 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 10 client.23203.objecter successfully canceled tick 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5a3c06c820 msgr2=0x7f5a3c06ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f5a3c06c820 0x7f5a3c06ece0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5a44001780 tx=0x7f5a44009040 comp rx=0 tx=0).stop 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 10 monclient: shutdown 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a5013f5c0 msgr2=0x7f5a5013a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 --2- 172.21.15.67:0/3437266736 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5a5013f5c0 0x7f5a5013a880 secure :-1 s=READY pgs=5900 cs=0 l=1 rev1=1 crypto rx=0x7f5a48009eb0 tx=0x7f5a48004a40 comp rx=0 tx=0).stop 2023-12-01T17:36:34.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a54ef7640 1 -- 172.21.15.67:0/3437266736 reap_dead start 2023-12-01T17:36:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 shutdown_connections 2023-12-01T17:36:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 >> 172.21.15.67:0/3437266736 conn(0x7f5a500b7870 msgr2=0x7f5a5013eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 shutdown_connections 2023-12-01T17:36:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 -- 172.21.15.67:0/3437266736 wait complete. 2023-12-01T17:36:34.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.842+0000 7f5a5695a640 1 librados: shutdown 2023-12-01T17:36:34.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-12-01T17:36:34.960 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.961+0000 7f1946aa2640 10 monclient: get_monmap_and_config 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.961+0000 7f1946aa2640 10 monclient: build_initial_monmap 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.961+0000 7f1946aa2640 10 monclient: monmap: 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:34.962129+0000 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:34.962129+0000 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:34.961 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:34.962 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:34.962 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.963 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 1 Processor -- start 2023-12-01T17:36:34.963 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 1 -- start start 2023-12-01T17:36:34.963 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient: init 2023-12-01T17:36:34.963 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:34.963 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-12-01T17:36:34.963 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f19400a8a20 0x7f19400a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f19400a8a20 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1940145ab0 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f1940145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f194013f800 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1940146430 con 0x7f1940145ab0 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1940145620 con 0x7f19400a8a20 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19400aace0 con 0x7f194013f800 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.962+0000 7f1946aa2640 10 monclient(hunting): authenticate will time out at 3326.200223s 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194503f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f1940145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194483e640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f19400a8a20 0x7f19400a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194503f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f1940145ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:45820/0 (socket says 172.21.15.67:45820) 2023-12-01T17:36:34.964 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194503f640 1 -- 172.21.15.67:0/2692347490 learned_addr learned my addr 172.21.15.67:0/2692347490 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194503f640 10 monclient(hunting): get_auth_request con 0x7f194013f800 auth_method 0 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194503f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194503f640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.963+0000 7f194503f640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194483e640 10 monclient(hunting): get_auth_request con 0x7f19400a8a20 auth_method 0 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194483e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194483e640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194483e640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194503f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194503f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.965 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194503f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194483e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194483e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.964+0000 7f194483e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 10 monclient(hunting): handle_auth_done global_id 17951 payload 274 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 1 -- 172.21.15.67:0/2692347490 >> v2:172.21.15.92:3300/0 conn(0x7f19400a8a20 msgr2=0x7f19400a8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 1 --2- 172.21.15.67:0/2692347490 >> v2:172.21.15.92:3300/0 conn(0x7f19400a8a20 0x7f19400a8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 1 -- 172.21.15.67:0/2692347490 >> v1:172.21.15.92:6789/0 conn(0x7f1940145ab0 legacy=0x7f194013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.966 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 10 monclient: _finish_hunting 0 2023-12-01T17:36:34.967 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 1 monclient: found mon.noname-b 2023-12-01T17:36:34.967 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:34.967 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:34.967 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.966+0000 7f194503f640 1 -- 172.21.15.67:0/2692347490 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19400aafc0 con 0x7f194013f800 2023-12-01T17:36:34.967 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.968+0000 7f194503f640 1 --2- 172.21.15.67:0/2692347490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f1940145ef0 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f1938009e50 tx=0x7f193802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a9a69dfb4fcf4e server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.972+0000 7f193f7fe640 1 -- 172.21.15.67:0/2692347490 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193802cb00 con 0x7f194013f800 2023-12-01T17:36:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.972+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: dump: 2023-12-01T17:36:34.972 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: _finish_auth 0 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 5 monclient: authenticate success, global_id 17951 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 1 -- 172.21.15.67:0/2692347490 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193802cc60 con 0x7f194013f800 2023-12-01T17:36:34.973 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 4 set_mon_vals no callback set 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 1 -- 172.21.15.67:0/2692347490 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380343f0 con 0x7f194013f800 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f193f7fe640 10 monclient: dump: 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:34.974 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:34.975 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:34.975 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.976 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 10 monclient: shutdown 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 -- 172.21.15.67:0/2692347490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 msgr2=0x7f1940145ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2692347490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f1940145ef0 secure :-1 s=READY pgs=3071 cs=0 l=1 rev1=1 crypto rx=0x7f1938009e50 tx=0x7f193802bf60 comp rx=0 tx=0).stop 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 -- 172.21.15.67:0/2692347490 shutdown_connections 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2692347490 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f1940145ef0 unknown :-1 s=CLOSED pgs=3071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2692347490 >> v2:172.21.15.92:3300/0 conn(0x7f19400a8a20 0x7f19400a8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 -- 172.21.15.67:0/2692347490 >> 172.21.15.67:0/2692347490 conn(0x7f19400b7890 msgr2=0x7f19400b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 -- 172.21.15.67:0/2692347490 shutdown_connections 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.973+0000 7f1946aa2640 1 -- 172.21.15.67:0/2692347490 wait complete. 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 10 monclient: build_initial_monmap 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 10 monclient: monmap: 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:34.977 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 1 librados: starting msgr at 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 1 librados: starting objecter 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 1 Processor -- start 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 1 -- start start 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 1 librados: setting wanted keys 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 1 librados: calling monclient init 2023-12-01T17:36:34.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.974+0000 7f1946aa2640 10 monclient: init 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19400a8a20 0x7f19401443a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f19400a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f19401386b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f194013f800 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1940145ab0 0x7f1940138bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f1940145ab0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1940137fb0 con 0x7f19400a8a20 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1940138290 con 0x7f1940145ab0 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1940144b00 con 0x7f194013f800 2023-12-01T17:36:34.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:34.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.975+0000 7f1946aa2640 10 monclient(hunting): authenticate will time out at 3326.213159s 2023-12-01T17:36:34.980 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:34.980+0000 7f193ffff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f19401386b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.015+0000 7f194503f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1940145ab0 0x7f1940138bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.016+0000 7f194503f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1940145ab0 0x7f1940138bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42710/0 (socket says 172.21.15.67:42710) 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.016+0000 7f194503f640 1 -- 172.21.15.67:0/2284846343 learned_addr learned my addr 172.21.15.67:0/2284846343 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.016+0000 7f194503f640 10 monclient(hunting): get_auth_request con 0x7f1940145ab0 auth_method 0 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.016+0000 7f194503f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.016+0000 7f194503f640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.016+0000 7f194503f640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.017+0000 7f194503f640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.017+0000 7f194503f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.017+0000 7f194503f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 10 monclient(hunting): handle_auth_done global_id 23209 payload 824 2023-12-01T17:36:35.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 1 -- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 msgr2=0x7f19401386b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f19401386b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 1 -- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19400a8a20 msgr2=0x7f19401443a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19400a8a20 0x7f19401443a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 10 monclient: _finish_hunting 0 2023-12-01T17:36:35.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 1 monclient: found mon.noname-b 2023-12-01T17:36:35.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:35.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.020+0000 7f194503f640 1 -- 172.21.15.67:0/2284846343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1938009b30 con 0x7f1940145ab0 2023-12-01T17:36:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.022+0000 7f194503f640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1940145ab0 0x7f1940138bf0 secure :-1 s=READY pgs=5902 cs=0 l=1 rev1=1 crypto rx=0x7f1938009e50 tx=0x7f1938004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.022+0000 7f193d7fa640 10 client.?.objecter ms_handle_connect 0x7f1940145ab0 2023-12-01T17:36:35.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.022+0000 7f193d7fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 1 -- 172.21.15.67:0/2284846343 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193803e450 con 0x7f1940145ab0 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: dump: 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.026 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: _finish_auth 0 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: _check_auth_tickets 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 1 -- 172.21.15.67:0/2284846343 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193803e5b0 con 0x7f1940145ab0 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 5 monclient: authenticate success, global_id 23209 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 1 -- 172.21.15.67:0/2284846343 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193803e960 con 0x7f1940145ab0 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 10 monclient: _renew_subs 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19400b14d0 con 0x7f1940145ab0 2023-12-01T17:36:35.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193effd640 4 set_mon_vals no callback set 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f193d7fa640 10 monclient: dump: 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.028 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 10 client.23209.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 10 monclient: _renew_subs 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1940137130 con 0x7f1940145ab0 2023-12-01T17:36:35.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.026+0000 7f1946aa2640 1 librados: init done 2023-12-01T17:36:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.027+0000 7f1946aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.027+0000 7f1946aa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.027+0000 7f1946aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.027+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194004e840 con 0x7f1940145ab0 2023-12-01T17:36:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.277+0000 7f193d7fa640 1 -- 172.21.15.67:0/2284846343 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f193803eac0 con 0x7f1940145ab0 2023-12-01T17:36:35.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.278+0000 7f193d7fa640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f192406c820 0x7f192406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.278+0000 7f194483e640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f192406c820 0x7f192406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.278+0000 7f194483e640 10 monclient: get_auth_request con 0x7f192406c820 auth_method 0 2023-12-01T17:36:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.278+0000 7f193d7fa640 10 client.23209.objecter ms_handle_connect 0x7f192406c820 2023-12-01T17:36:35.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.278+0000 7f194483e640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f192406c820 0x7f192406ece0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1934006a90 tx=0x7f1934006a20 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.280+0000 7f193d7fa640 1 -- 172.21.15.67:0/2284846343 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f193807bfb0 con 0x7f1940145ab0 2023-12-01T17:36:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.281+0000 7f193d7fa640 10 client.23209.objecter ms_dispatch 0x7f1940000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.281+0000 7f193d7fa640 3 client.23209.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.281+0000 7f193d7fa640 3 client.23209.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:35.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.281+0000 7f193d7fa640 20 client.23209.objecter dump_active .. 0 homeless 2023-12-01T17:36:35.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.290+0000 7f193d7fa640 1 -- 172.21.15.67:0/2284846343 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1938077bf0 con 0x7f1940145ab0 2023-12-01T17:36:35.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.290+0000 7f193d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:35.290 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.290+0000 7f193d7fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:35.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.513+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1940138120 con 0x7f192406c820 2023-12-01T17:36:35.513 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.513+0000 7f193d7fa640 1 -- 172.21.15.67:0/2284846343 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1940138120 con 0x7f192406c820 2023-12-01T17:36:35.513 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 4 to deep-scrub 2023-12-01T17:36:35.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 10 librados: watch_flush enter 2023-12-01T17:36:35.515 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 10 librados: watch_flush exit 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 20 client.23209.objecter shutdown clearing up homeless session... 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 10 client.23209.objecter successfully canceled tick 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f192406c820 msgr2=0x7f192406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f192406c820 0x7f192406ece0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f1934006a90 tx=0x7f1934006a20 comp rx=0 tx=0).stop 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 10 monclient: shutdown 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1940145ab0 msgr2=0x7f1940138bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.515+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1940145ab0 0x7f1940138bf0 secure :-1 s=READY pgs=5902 cs=0 l=1 rev1=1 crypto rx=0x7f1938009e50 tx=0x7f1938004a40 comp rx=0 tx=0).stop 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 shutdown_connections 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1940145ab0 0x7f1940138bf0 unknown :-1 s=CLOSED pgs=5902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f192406c820 0x7f192406ece0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f194013f800 0x7f19401386b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.516 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 --2- 172.21.15.67:0/2284846343 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19400a8a20 0x7f19401443a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 >> 172.21.15.67:0/2284846343 conn(0x7f19400b7890 msgr2=0x7f194013e2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:35.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 shutdown_connections 2023-12-01T17:36:35.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 -- 172.21.15.67:0/2284846343 wait complete. 2023-12-01T17:36:35.517 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.516+0000 7f1946aa2640 1 librados: shutdown 2023-12-01T17:36:35.526 INFO:tasks.ceph:Scrubbing osd.5 2023-12-01T17:36:35.526 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:35.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.637+0000 7fe40751d640 10 monclient: get_monmap_and_config 2023-12-01T17:36:35.636 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.637+0000 7fe40751d640 10 monclient: build_initial_monmap 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.637+0000 7fe40751d640 10 monclient: monmap: 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:35.638076+0000 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:35.638076+0000 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:35.637 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.637+0000 7fe40751d640 1 Processor -- start 2023-12-01T17:36:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 1 -- start start 2023-12-01T17:36:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient: init 2023-12-01T17:36:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-12-01T17:36:35.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fe400071690 0x7fe400145620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient(hunting): picked mon.noname-c con 0x7fe400071690 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe400145b60 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe40006cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient(hunting): picked mon.noname-b con 0x7fe400145f80 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40006bc90 con 0x7fe400145b60 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe400070760 con 0x7fe400071690 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe400071a70 con 0x7fe400145f80 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.638+0000 7fe40751d640 10 monclient(hunting): authenticate will time out at 3326.876180s 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe405aba640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe40006cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe405aba640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe40006cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:45828/0 (socket says 172.21.15.67:45828) 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe405aba640 1 -- 172.21.15.67:0/4024897406 learned_addr learned my addr 172.21.15.67:0/4024897406 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:35.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe405aba640 10 monclient(hunting): get_auth_request con 0x7fe400145f80 auth_method 0 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe405aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe405aba640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe405aba640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.639+0000 7fe4052b9640 1 --2- 172.21.15.67:0/4024897406 >> v2:172.21.15.92:3300/0 conn(0x7fe400071690 0x7fe400145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe4052b9640 10 monclient(hunting): get_auth_request con 0x7fe400071690 auth_method 0 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe4052b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe4052b9640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe4052b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe405aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe405aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe405aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe4052b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe4052b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.640+0000 7fe4052b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.641+0000 7fe405aba640 10 monclient(hunting): handle_auth_done global_id 17954 payload 274 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.641+0000 7fe405aba640 1 -- 172.21.15.67:0/4024897406 >> v2:172.21.15.92:3300/0 conn(0x7fe400071690 msgr2=0x7fe400145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.641+0000 7fe405aba640 1 --2- 172.21.15.67:0/4024897406 >> v2:172.21.15.92:3300/0 conn(0x7fe400071690 0x7fe400145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.641+0000 7fe405aba640 1 -- 172.21.15.67:0/4024897406 >> v1:172.21.15.92:6789/0 conn(0x7fe400145b60 legacy=0x7fe40006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.641+0000 7fe405aba640 10 monclient: _finish_hunting 0 2023-12-01T17:36:35.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.641+0000 7fe405aba640 1 monclient: found mon.noname-b 2023-12-01T17:36:35.643 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.641+0000 7fe405aba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:35.643 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.642+0000 7fe405aba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:35.643 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.642+0000 7fe405aba640 1 -- 172.21.15.67:0/4024897406 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe400070940 con 0x7fe400145f80 2023-12-01T17:36:35.643 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.643+0000 7fe405aba640 1 --2- 172.21.15.67:0/4024897406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe40006cb80 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009870 tx=0x7fe3fc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4eaaf010b6432d8f server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 1 -- 172.21.15.67:0/4024897406 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc030840 con 0x7fe400145f80 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: dump: 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.647 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: _finish_auth 0 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 1 -- 172.21.15.67:0/4024897406 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc0309a0 con 0x7fe400145f80 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 1 -- 172.21.15.67:0/4024897406 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc030d30 con 0x7fe400145f80 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe40751d640 5 monclient: authenticate success, global_id 17954 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe40751d640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe40751d640 4 set_mon_vals no callback set 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.647+0000 7fe3f7fff640 10 monclient: dump: 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.648 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.649 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 10 monclient: shutdown 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 -- 172.21.15.67:0/4024897406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 msgr2=0x7fe40006cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 --2- 172.21.15.67:0/4024897406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe40006cb80 secure :-1 s=READY pgs=3072 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009870 tx=0x7fe3fc02b980 comp rx=0 tx=0).stop 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 -- 172.21.15.67:0/4024897406 shutdown_connections 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 --2- 172.21.15.67:0/4024897406 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe40006cb80 unknown :-1 s=CLOSED pgs=3072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 --2- 172.21.15.67:0/4024897406 >> v2:172.21.15.92:3300/0 conn(0x7fe400071690 0x7fe400145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 -- 172.21.15.67:0/4024897406 >> 172.21.15.67:0/4024897406 conn(0x7fe4000b5830 msgr2=0x7fe4000b5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 -- 172.21.15.67:0/4024897406 shutdown_connections 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.648+0000 7fe40751d640 1 -- 172.21.15.67:0/4024897406 wait complete. 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient: build_initial_monmap 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient: monmap: 2023-12-01T17:36:35.653 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 librados: starting msgr at 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 librados: starting objecter 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 Processor -- start 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 -- start start 2023-12-01T17:36:35.654 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 librados: setting wanted keys 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 librados: calling monclient init 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient: init 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe400071690 0x7fe400144a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): picked mon.noname-a con 0x7fe400071690 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe400145b60 0x7fe400142020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): picked mon.noname-b con 0x7fe400145b60 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe4001da210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): picked mon.noname-c con 0x7fe400145f80 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe400142bd0 con 0x7fe400071690 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4001426b0 con 0x7fe400145b60 2023-12-01T17:36:35.655 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4001415f0 con 0x7fe400145f80 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.649+0000 7fe40751d640 10 monclient(hunting): authenticate will time out at 3326.887381s 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe405aba640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe4001da210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe404ab8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe400145b60 0x7fe400142020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe4052b9640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe400071690 0x7fe400144a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe404ab8640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe400145b60 0x7fe400142020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42726/0 (socket says 172.21.15.67:42726) 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe404ab8640 1 -- 172.21.15.67:0/2180204619 learned_addr learned my addr 172.21.15.67:0/2180204619 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe404ab8640 10 monclient(hunting): get_auth_request con 0x7fe400145b60 auth_method 0 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe404ab8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe404ab8640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe404ab8640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe405aba640 10 monclient(hunting): get_auth_request con 0x7fe400145f80 auth_method 0 2023-12-01T17:36:35.656 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe405aba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe405aba640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.650+0000 7fe405aba640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.651+0000 7fe404ab8640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.651+0000 7fe404ab8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.651+0000 7fe404ab8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.651+0000 7fe405aba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.651+0000 7fe405aba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.651+0000 7fe405aba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.652+0000 7fe4052b9640 10 monclient(hunting): get_auth_request con 0x7fe400071690 auth_method 0 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.652+0000 7fe4052b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.652+0000 7fe4052b9640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.652+0000 7fe4052b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe4052b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe4052b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.657 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe4052b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 10 monclient(hunting): handle_auth_done global_id 23218 payload 824 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 1 -- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 msgr2=0x7fe4001da210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fe400145f80 0x7fe4001da210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 1 -- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe400071690 msgr2=0x7fe400144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe400071690 0x7fe400144a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 10 monclient: _finish_hunting 0 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 1 monclient: found mon.noname-b 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.658 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.653+0000 7fe404ab8640 1 -- 172.21.15.67:0/2180204619 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc0094d0 con 0x7fe400145b60 2023-12-01T17:36:35.659 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.655+0000 7fe404ab8640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe400145b60 0x7fe400142020 secure :-1 s=READY pgs=5904 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c400 tx=0x7fe3f000c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.655+0000 7fe3f5ffb640 10 client.?.objecter ms_handle_connect 0x7fe400145b60 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.656+0000 7fe3f5ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f00052d0 con 0x7fe400145b60 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: dump: 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.660 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f0005450 con 0x7fe400145b60 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0007260 con 0x7fe400145b60 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f5ffb640 10 monclient: dump: 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.661 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 5 monclient: authenticate success, global_id 23218 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 10 monclient: _renew_subs 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe400141810 con 0x7fe400145b60 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe3f77fe640 4 set_mon_vals no callback set 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 10 client.23218.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 10 monclient: _renew_subs 2023-12-01T17:36:35.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4001da930 con 0x7fe400145b60 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.659+0000 7fe40751d640 1 librados: init done 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.660+0000 7fe40751d640 10 client.23218.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.660+0000 7fe40751d640 15 client.23218.objecter _session_command_op_assign -1 1 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.660+0000 7fe40751d640 20 client.23218.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.660+0000 7fe40751d640 10 client.23218.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.660+0000 7fe40751d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.660+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe40006bee0 con 0x7fe400145b60 2023-12-01T17:36:35.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.678+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fe3f00073c0 con 0x7fe400145b60 2023-12-01T17:36:35.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.678+0000 7fe3f5ffb640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe3e406c820 0x7fe3e406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.678+0000 7fe4052b9640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe3e406c820 0x7fe3e406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.678+0000 7fe4052b9640 10 monclient: get_auth_request con 0x7fe3e406c820 auth_method 0 2023-12-01T17:36:35.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.679+0000 7fe3f5ffb640 10 client.23218.objecter ms_handle_connect 0x7fe3e406c820 2023-12-01T17:36:35.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.679+0000 7fe4052b9640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe3e406c820 0x7fe3e406ece0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8011340 tx=0x7fe3f800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fe3f0007670 con 0x7fe400145b60 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 10 client.23218.objecter ms_dispatch 0x7fe400000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 3 client.23218.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 3 client.23218.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 10 client.23218.objecter checking command 1 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fe3e4074380 0x7fe3e4076820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 20 client.23218.objecter _get_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 20 client.23218.objecter put_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 15 client.23218.objecter _session_command_op_remove -1 1 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 20 client.23218.objecter _get_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 20 client.23218.objecter get_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 15 client.23218.objecter _session_command_op_assign 5 1 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 20 client.23218.objecter put_session s=0x7fe3e4073cb0 osd=5 4 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 10 client.23218.objecter _send_command 1 2023-12-01T17:36:35.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 --> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe3e4076ef0 con 0x7fe3e4074380 2023-12-01T17:36:35.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.681+0000 7fe3f5ffb640 20 client.23218.objecter dump_active .. 0 homeless 2023-12-01T17:36:35.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.682+0000 7fe405aba640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fe3e4074380 0x7fe3e4076820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.682+0000 7fe405aba640 10 monclient: get_auth_request con 0x7fe3e4074380 auth_method 0 2023-12-01T17:36:35.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.684+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=1099) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3f00546b0 con 0x7fe400145b60 2023-12-01T17:36:35.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.684+0000 7fe3f5ffb640 10 monclient: handle_get_version_reply finishing 1 version 1099 2023-12-01T17:36:35.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.685+0000 7fe405aba640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fe3e4074380 0x7fe3e4076820 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.685+0000 7fe3f5ffb640 10 client.23218.objecter ms_handle_connect 0x7fe3e4074380 2023-12-01T17:36:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.708+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== osd.5 v2:172.21.15.92:6800/1636405563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe3e4076ef0 con 0x7fe3e4074380 2023-12-01T17:36:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.708+0000 7fe3f5ffb640 10 client.23218.objecter ms_dispatch 0x7fe400000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:35.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.708+0000 7fe3f5ffb640 10 client.23218.objecter _finish_command 1 = system:0 2023-12-01T17:36:35.708 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.708+0000 7fe3f5ffb640 20 client.23218.objecter put_session s=0x7fe3e4073cb0 osd=5 4 2023-12-01T17:36:35.708 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.708+0000 7fe3f5ffb640 15 client.23218.objecter _session_command_op_remove 5 1 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 10 client.23218.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 15 client.23218.objecter _session_command_op_assign -1 2 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 20 client.23218.objecter _get_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 20 client.23218.objecter put_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 20 client.23218.objecter _get_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 15 client.23218.objecter _session_command_op_remove -1 2 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 20 client.23218.objecter get_session s=0x7fe3e4073cb0 osd=5 3 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 15 client.23218.objecter _session_command_op_assign 5 2 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 20 client.23218.objecter put_session s=0x7fe3e4073cb0 osd=5 4 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 10 client.23218.objecter _send_command 2 2023-12-01T17:36:35.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.721+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 --> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe400070cb0 con 0x7fe3e4074380 2023-12-01T17:36:35.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe3f5ffb640 1 -- 172.21.15.67:0/2180204619 <== osd.5 v2:172.21.15.92:6800/1636405563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe3fc030b10 con 0x7fe3e4074380 2023-12-01T17:36:35.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe3f5ffb640 10 client.23218.objecter ms_dispatch 0x7fe400000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:35.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe3f5ffb640 10 client.23218.objecter _finish_command 2 = system:0 2023-12-01T17:36:35.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe3f5ffb640 20 client.23218.objecter put_session s=0x7fe3e4073cb0 osd=5 4 2023-12-01T17:36:35.734 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe3f5ffb640 15 client.23218.objecter _session_command_op_remove 5 2 2023-12-01T17:36:35.735 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:35.735 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:35.735 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:35.736 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 10 librados: watch_flush enter 2023-12-01T17:36:35.736 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 10 librados: watch_flush exit 2023-12-01T17:36:35.736 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 10 client.23218.objecter close_session for osd.5 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fe3e4074380 msgr2=0x7fe3e4076820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:6800/1636405563,v1:172.21.15.92:6801/1636405563] conn(0x7fe3e4074380 0x7fe3e4076820 crc :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 20 client.23218.objecter put_session s=0x7fe3e4073cb0 osd=5 1 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 20 client.23218.objecter shutdown clearing up homeless session... 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 10 client.23218.objecter successfully canceled tick 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe3e406c820 msgr2=0x7fe3e406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fe3e406c820 0x7fe3e406ece0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fe3f8011340 tx=0x7fe3f800b040 comp rx=0 tx=0).stop 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 10 monclient: shutdown 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe400145b60 msgr2=0x7fe400142020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 --2- 172.21.15.67:0/2180204619 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fe400145b60 0x7fe400142020 secure :-1 s=READY pgs=5904 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c400 tx=0x7fe3f000c8c0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe405aba640 1 -- 172.21.15.67:0/2180204619 reap_dead start 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 shutdown_connections 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 >> 172.21.15.67:0/2180204619 conn(0x7fe4000b5830 msgr2=0x7fe40013be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:35.737 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 shutdown_connections 2023-12-01T17:36:35.738 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 -- 172.21.15.67:0/2180204619 wait complete. 2023-12-01T17:36:35.738 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.735+0000 7fe40751d640 1 librados: shutdown 2023-12-01T17:36:35.745 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-12-01T17:36:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.855+0000 7f443a64f640 10 monclient: get_monmap_and_config 2023-12-01T17:36:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.855+0000 7f443a64f640 10 monclient: build_initial_monmap 2023-12-01T17:36:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.855+0000 7f443a64f640 10 monclient: monmap: 2023-12-01T17:36:35.854 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:35.855848+0000 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:35.855848+0000 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:35.855 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.855+0000 7f443a64f640 1 Processor -- start 2023-12-01T17:36:35.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.855+0000 7f443a64f640 1 -- start start 2023-12-01T17:36:35.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.855+0000 7f443a64f640 10 monclient: init 2023-12-01T17:36:35.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:35.857 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44340a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f44340a8e00 0x7f443413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient(hunting): picked mon.noname-a con 0x7f44340a8e00 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44341423b0 0x7f4434142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient(hunting): picked mon.noname-b con 0x7f44341423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44340b21f0 con 0x7f44340a8a20 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44340a9cc0 con 0x7f44340a8e00 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44340a9ed0 con 0x7f44341423b0 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.856+0000 7f443a64f640 10 monclient(hunting): authenticate will time out at 3327.094026s 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f4433fff640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f44340a8a20 0x7f443413eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:36976/0 (socket says 172.21.15.67:36976) 2023-12-01T17:36:35.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f4433fff640 1 -- 172.21.15.67:0/1277805281 learned_addr learned my addr 172.21.15.67:0/1277805281 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f44337fe640 1 --2- 172.21.15.67:0/1277805281 >> v2:172.21.15.67:3300/0 conn(0x7f44340a8e00 0x7f443413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f4438bec640 1 --2- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44341423b0 0x7f4434142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f44337fe640 10 monclient(hunting): get_auth_request con 0x7f44340a8e00 auth_method 0 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f44337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f44337fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f44337fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f4438bec640 10 monclient(hunting): get_auth_request con 0x7f44341423b0 auth_method 0 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f4438bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f4438bec640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.857+0000 7f4438bec640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.858+0000 7f44337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.858+0000 7f44337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.858+0000 7f44337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.858+0000 7f4438bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.858+0000 7f4438bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.858+0000 7f4438bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f4432ffd640 1 -- 172.21.15.67:0/1277805281 <== mon.1 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32255294 0 0) 0x7f44340b21f0 con 0x7f44340a8a20 2023-12-01T17:36:35.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f4432ffd640 10 monclient(hunting): _negotiate 2023-12-01T17:36:35.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f4432ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f4432ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f4432ffd640 10 monclient(hunting): my global_id is 23221 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f4432ffd640 1 -- 172.21.15.67:0/1277805281 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f441c003520 con 0x7f44340a8a20 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 10 monclient(hunting): handle_auth_done global_id 16026 payload 274 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 1 -- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44341423b0 msgr2=0x7f4434142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 1 --2- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44341423b0 0x7f4434142840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 1 -- 172.21.15.67:0/1277805281 >> v1:172.21.15.92:6789/0 conn(0x7f44340a8a20 legacy=0x7f443413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 10 monclient: _finish_hunting 0 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 1 monclient: found mon.noname-a 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:36:35.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.859+0000 7f44337fe640 1 -- 172.21.15.67:0/1277805281 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44340aac70 con 0x7f44340a8e00 2023-12-01T17:36:35.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.862+0000 7f44337fe640 1 --2- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44340a8e00 0x7f443413f5c0 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7f44280097a0 tx=0x7f442802b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925b615b913e589e server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 1 -- 172.21.15.67:0/1277805281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442802f050 con 0x7f44340a8e00 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: dump: 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.865 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: _finish_auth 0 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 1 -- 172.21.15.67:0/1277805281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4428031700 con 0x7f44340a8e00 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 1 -- 172.21.15.67:0/1277805281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428031a90 con 0x7f44340a8e00 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f4432ffd640 10 monclient: dump: 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.866 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f443a64f640 5 monclient: authenticate success, global_id 16026 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f443a64f640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:35.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.865+0000 7f443a64f640 4 set_mon_vals no callback set 2023-12-01T17:36:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 10 monclient: shutdown 2023-12-01T17:36:35.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 -- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44340a8e00 msgr2=0x7f443413f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 --2- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44340a8e00 0x7f443413f5c0 secure :-1 s=READY pgs=3173 cs=0 l=1 rev1=1 crypto rx=0x7f44280097a0 tx=0x7f442802b7a0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 -- 172.21.15.67:0/1277805281 shutdown_connections 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 --2- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44341423b0 0x7f4434142840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 --2- 172.21.15.67:0/1277805281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44340a8e00 0x7f443413f5c0 unknown :-1 s=CLOSED pgs=3173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 -- 172.21.15.67:0/1277805281 >> 172.21.15.67:0/1277805281 conn(0x7f44340b7870 msgr2=0x7f44340b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 -- 172.21.15.67:0/1277805281 shutdown_connections 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.866+0000 7f443a64f640 1 -- 172.21.15.67:0/1277805281 wait complete. 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 10 monclient: build_initial_monmap 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 10 monclient: monmap: 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:35.871 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 1 librados: starting msgr at 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 1 librados: starting objecter 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 1 Processor -- start 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 1 -- start start 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 1 librados: setting wanted keys 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 1 librados: calling monclient init 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.867+0000 7f443a64f640 10 monclient: init 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:35.872 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient: _add_conns ranks=[1,2,0] 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44340a8a20 0x7f4434146910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): picked mon.noname-b con 0x7f44340a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44340a8e00 0x7f44341daa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): picked mon.noname-c con 0x7f44340a8e00 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44341423b0 0x7f44341daf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): picked mon.noname-a con 0x7f44341423b0 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44341da500 con 0x7f44341423b0 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4434143390 con 0x7f44340a8a20 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4434143550 con 0x7f44340a8e00 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f443a64f640 10 monclient(hunting): authenticate will time out at 3327.105669s 2023-12-01T17:36:35.873 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f4438bec640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44341423b0 0x7f44341daf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f44337fe640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44340a8e00 0x7f44341daa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.868+0000 7f4433fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44340a8a20 0x7f4434146910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44340a8a20 0x7f4434146910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42734/0 (socket says 172.21.15.67:42734) 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 1 -- 172.21.15.67:0/3373787949 learned_addr learned my addr 172.21.15.67:0/3373787949 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 10 monclient(hunting): get_auth_request con 0x7f44340a8a20 auth_method 0 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4438bec640 10 monclient(hunting): get_auth_request con 0x7f44341423b0 auth_method 0 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4438bec640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4438bec640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4438bec640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f44337fe640 10 monclient(hunting): get_auth_request con 0x7f44340a8e00 auth_method 0 2023-12-01T17:36:35.874 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f44337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f44337fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f44337fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4433fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4438bec640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4438bec640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f4438bec640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f44337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f44337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:35.875 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.869+0000 7f44337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 10 monclient(hunting): handle_auth_done global_id 23224 payload 824 2023-12-01T17:36:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 1 -- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44340a8e00 msgr2=0x7f44341daa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44340a8e00 0x7f44341daa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 1 -- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44341423b0 msgr2=0x7f44341daf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44341423b0 0x7f44341daf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:35.876 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 10 monclient: _finish_hunting 0 2023-12-01T17:36:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 1 monclient: found mon.noname-b 2023-12-01T17:36:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.872+0000 7f4433fff640 1 -- 172.21.15.67:0/3373787949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4428009400 con 0x7f44340a8a20 2023-12-01T17:36:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.874+0000 7f4433fff640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44340a8a20 0x7f4434146910 secure :-1 s=READY pgs=5906 cs=0 l=1 rev1=1 crypto rx=0x7f442400c500 tx=0x7f442400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.874+0000 7f4430ff9640 10 client.?.objecter ms_handle_connect 0x7f44340a8a20 2023-12-01T17:36:35.877 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.874+0000 7f4430ff9640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 1 -- 172.21.15.67:0/3373787949 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424005ce0 con 0x7f44340a8a20 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: dump: 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.878 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: _finish_auth 0 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: _check_auth_tickets 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 1 -- 172.21.15.67:0/3373787949 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4424005e40 con 0x7f44340a8a20 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 5 monclient: authenticate success, global_id 23224 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 10 monclient: _renew_subs 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4434145b70 con 0x7f44340a8a20 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 1 -- 172.21.15.67:0/3373787949 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4424007330 con 0x7f44340a8a20 2023-12-01T17:36:35.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f4430ff9640 10 monclient: dump: 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 10 client.23224.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 10 monclient: _renew_subs 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4434146050 con 0x7f44340a8a20 2023-12-01T17:36:35.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f44327fc640 4 set_mon_vals no callback set 2023-12-01T17:36:35.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.878+0000 7f443a64f640 1 librados: init done 2023-12-01T17:36:35.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.879+0000 7f443a64f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:35.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.879+0000 7f443a64f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:35.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.879+0000 7f443a64f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:35.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.879+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44340b67f0 con 0x7f44340a8a20 2023-12-01T17:36:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.897+0000 7f4430ff9640 1 -- 172.21.15.67:0/3373787949 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f4424007550 con 0x7f44340a8a20 2023-12-01T17:36:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.897+0000 7f4430ff9640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f442006c820 0x7f442006ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.898+0000 7f44337fe640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f442006c820 0x7f442006ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.898+0000 7f44337fe640 10 monclient: get_auth_request con 0x7f442006c820 auth_method 0 2023-12-01T17:36:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.898+0000 7f4430ff9640 10 client.23224.objecter ms_handle_connect 0x7f442006c820 2023-12-01T17:36:35.897 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:35.898+0000 7f44337fe640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f442006c820 0x7f442006ece0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4428035000 tx=0x7f4428008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.119+0000 7f4430ff9640 1 -- 172.21.15.67:0/3373787949 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f4424057f80 con 0x7f44340a8a20 2023-12-01T17:36:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.119+0000 7f4430ff9640 10 client.23224.objecter ms_dispatch 0x7f4434000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:36.118 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.119+0000 7f4430ff9640 3 client.23224.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.119+0000 7f4430ff9640 3 client.23224.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:36.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.119+0000 7f4430ff9640 20 client.23224.objecter dump_active .. 0 homeless 2023-12-01T17:36:36.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.128+0000 7f4430ff9640 1 -- 172.21.15.67:0/3373787949 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44240542c0 con 0x7f44340a8a20 2023-12-01T17:36:36.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.128+0000 7f4430ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:36.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.128+0000 7f4430ff9640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.313+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f443404e840 con 0x7f442006c820 2023-12-01T17:36:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.313+0000 7f4430ff9640 1 -- 172.21.15.67:0/3373787949 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f443404e840 con 0x7f442006c820 2023-12-01T17:36:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 5 to deep-scrub 2023-12-01T17:36:36.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 10 librados: watch_flush enter 2023-12-01T17:36:36.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 10 librados: watch_flush exit 2023-12-01T17:36:36.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 20 client.23224.objecter shutdown clearing up homeless session... 2023-12-01T17:36:36.315 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 10 client.23224.objecter successfully canceled tick 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f442006c820 msgr2=0x7f442006ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f442006c820 0x7f442006ece0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f4428035000 tx=0x7f4428008030 comp rx=0 tx=0).stop 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 10 monclient: shutdown 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44340a8a20 msgr2=0x7f4434146910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44340a8a20 0x7f4434146910 secure :-1 s=READY pgs=5906 cs=0 l=1 rev1=1 crypto rx=0x7f442400c500 tx=0x7f442400c9c0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 shutdown_connections 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f44341423b0 0x7f44341daf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f442006c820 0x7f442006ece0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f44340a8e00 0x7f44341daa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 --2- 172.21.15.67:0/3373787949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f44340a8a20 0x7f4434146910 unknown :-1 s=CLOSED pgs=5906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.315+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 >> 172.21.15.67:0/3373787949 conn(0x7f44340b7870 msgr2=0x7f443413bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.316+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 shutdown_connections 2023-12-01T17:36:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.316+0000 7f443a64f640 1 -- 172.21.15.67:0/3373787949 wait complete. 2023-12-01T17:36:36.317 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.316+0000 7f443a64f640 1 librados: shutdown 2023-12-01T17:36:36.326 INFO:tasks.ceph:Scrubbing osd.6 2023-12-01T17:36:36.326 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.438+0000 7f20cd061640 10 monclient: get_monmap_and_config 2023-12-01T17:36:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.438+0000 7f20cd061640 10 monclient: build_initial_monmap 2023-12-01T17:36:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.438+0000 7f20cd061640 10 monclient: monmap: 2023-12-01T17:36:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:36.439603+0000 2023-12-01T17:36:36.439 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:36.439603+0000 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.439+0000 7f20cd061640 1 Processor -- start 2023-12-01T17:36:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.439+0000 7f20cd061640 1 -- start start 2023-12-01T17:36:36.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.439+0000 7f20cd061640 10 monclient: init 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20c80a8a20 0x7f20c813eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient(hunting): picked mon.noname-c con 0x7f20c80a8a20 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c813f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient(hunting): picked mon.noname-b con 0x7f20c80a8e00 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f20c81423b0 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:36.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c80b21f0 con 0x7f20c81423b0 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c80a9cc0 con 0x7f20c80a8a20 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8142f50 con 0x7f20c80a8e00 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20cd061640 10 monclient(hunting): authenticate will time out at 3327.677647s 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20c6d9d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f20c80a8a20 0x7f20c813eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20c659c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c813f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20c759e640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f20c81423b0 0x7f20c8142840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56726/0 (socket says 172.21.15.67:56726) 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.440+0000 7f20c759e640 1 -- 172.21.15.67:0/2296126 learned_addr learned my addr 172.21.15.67:0/2296126 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c6d9d640 10 monclient(hunting): get_auth_request con 0x7f20c80a8a20 auth_method 0 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c6d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c659c640 10 monclient(hunting): get_auth_request con 0x7f20c80a8e00 auth_method 0 2023-12-01T17:36:36.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c659c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.441+0000 7f20c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 10 monclient(hunting): handle_auth_done global_id 23227 payload 274 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 1 -- 172.21.15.67:0/2296126 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 msgr2=0x7f20c813f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 1 --2- 172.21.15.67:0/2296126 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c813f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 1 -- 172.21.15.67:0/2296126 >> v1:172.21.15.67:6789/0 conn(0x7f20c81423b0 legacy=0x7f20c8142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 10 monclient: _finish_hunting 0 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 1 monclient: found mon.noname-c 2023-12-01T17:36:36.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:36.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.443+0000 7f20c6d9d640 1 -- 172.21.15.67:0/2296126 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c813f930 con 0x7f20c80a8a20 2023-12-01T17:36:36.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.445+0000 7f20c6d9d640 1 --2- 172.21.15.67:0/2296126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80a8a20 0x7f20c813eeb0 secure :-1 s=READY pgs=5907 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009d90 tx=0x7f20b802bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f53a7d6bd1198e0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 1 -- 172.21.15.67:0/2296126 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b802cb30 con 0x7f20c80a8a20 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: dump: 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.449 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: _finish_auth 0 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: _check_auth_tickets 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 1 -- 172.21.15.67:0/2296126 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b802cc90 con 0x7f20c80a8a20 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 1 -- 172.21.15.67:0/2296126 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b8034440 con 0x7f20c80a8a20 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:36.450 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20c5d9b640 10 monclient: dump: 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20cd061640 5 monclient: authenticate success, global_id 23227 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20cd061640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.448+0000 7f20cd061640 4 set_mon_vals no callback set 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 10 monclient: shutdown 2023-12-01T17:36:36.451 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 -- 172.21.15.67:0/2296126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80a8a20 msgr2=0x7f20c813eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 --2- 172.21.15.67:0/2296126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80a8a20 0x7f20c813eeb0 secure :-1 s=READY pgs=5907 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009d90 tx=0x7f20b802bd80 comp rx=0 tx=0).stop 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 -- 172.21.15.67:0/2296126 shutdown_connections 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 --2- 172.21.15.67:0/2296126 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c813f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 --2- 172.21.15.67:0/2296126 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c80a8a20 0x7f20c813eeb0 unknown :-1 s=CLOSED pgs=5907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 -- 172.21.15.67:0/2296126 >> 172.21.15.67:0/2296126 conn(0x7f20c80b7870 msgr2=0x7f20c80b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 -- 172.21.15.67:0/2296126 shutdown_connections 2023-12-01T17:36:36.452 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.449+0000 7f20cd061640 1 -- 172.21.15.67:0/2296126 wait complete. 2023-12-01T17:36:36.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 10 monclient: build_initial_monmap 2023-12-01T17:36:36.456 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 10 monclient: monmap: 2023-12-01T17:36:36.456 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:36.457 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:36.457 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:36.457 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:36.457 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:36.457 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 1 librados: starting msgr at 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 1 librados: starting objecter 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 1 Processor -- start 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 1 -- start start 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 1 librados: setting wanted keys 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 1 librados: calling monclient init 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.450+0000 7f20cd061640 10 monclient: init 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c80a8a20 0x7f20c8145010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): picked mon.noname-a con 0x7f20c80a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c8143600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): picked mon.noname-c con 0x7f20c80a8e00 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c81423b0 0x7f20c8137b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): picked mon.noname-b con 0x7f20c81423b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c81d7190 con 0x7f20c80a8a20 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c8144bf0 con 0x7f20c81423b0 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c81385e0 con 0x7f20c80a8e00 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.451+0000 7f20cd061640 10 monclient(hunting): authenticate will time out at 3327.688883s 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c8143600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c6d9d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c80a8a20 0x7f20c8145010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c81423b0 0x7f20c8137b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c8143600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:45878/0 (socket says 172.21.15.67:45878) 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c81423b0 0x7f20c8137b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42746/0 (socket says 172.21.15.67:42746) 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 1 -- 172.21.15.67:0/3253050648 learned_addr learned my addr 172.21.15.67:0/3253050648 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 10 monclient(hunting): get_auth_request con 0x7f20c80a8e00 auth_method 0 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 10 monclient(hunting): get_auth_request con 0x7f20c81423b0 auth_method 0 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c6d9d640 10 monclient(hunting): get_auth_request con 0x7f20c80a8a20 auth_method 0 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c6d9d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c6d9d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.452+0000 7f20c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.453+0000 7f20c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.453+0000 7f20c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.453+0000 7f20c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 10 monclient(hunting): handle_auth_done global_id 23230 payload 824 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 1 -- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 msgr2=0x7f20c8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f20c80a8e00 0x7f20c8143600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 1 -- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c80a8a20 msgr2=0x7f20c8145010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f20c80a8a20 0x7f20c8145010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 10 monclient: _finish_hunting 0 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 1 monclient: found mon.noname-b 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.455+0000 7f20c759e640 1 -- 172.21.15.67:0/3253050648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20b8009a70 con 0x7f20c81423b0 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.457+0000 7f20c759e640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c81423b0 0x7f20c8137b40 secure :-1 s=READY pgs=5908 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c3c0 tx=0x7f20c000c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.457+0000 7f20bf7fe640 10 client.?.objecter ms_handle_connect 0x7f20c81423b0 2023-12-01T17:36:36.462 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.457+0000 7f20bf7fe640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:36.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c0005ce0 con 0x7f20c81423b0 2023-12-01T17:36:36.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:36.463 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: dump: 2023-12-01T17:36:36.463 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: _finish_auth 0 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20c0005e40 con 0x7f20c81423b0 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 5 monclient: authenticate success, global_id 23230 2023-12-01T17:36:36.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 10 monclient: _renew_subs 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c81387b0 con 0x7f20c81423b0 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20c00072f0 con 0x7f20c81423b0 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20bf7fe640 10 monclient: dump: 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.465 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20c559a640 4 set_mon_vals no callback set 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 10 client.23230.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 10 monclient: _renew_subs 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c81d7910 con 0x7f20c81423b0 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.460+0000 7f20cd061640 1 librados: init done 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.461+0000 7f20cd061640 10 client.23230.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.461+0000 7f20cd061640 15 client.23230.objecter _session_command_op_assign -1 1 2023-12-01T17:36:36.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.461+0000 7f20cd061640 20 client.23230.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.461+0000 7f20cd061640 10 client.23230.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.461+0000 7f20cd061640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.461+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f20c80aaf50 con 0x7f20c81423b0 2023-12-01T17:36:36.479 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.480+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f20c0007450 con 0x7f20c81423b0 2023-12-01T17:36:36.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.480+0000 7f20bf7fe640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f20a806c870 0x7f20a806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.480+0000 7f20c6d9d640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f20a806c870 0x7f20a806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.480+0000 7f20c6d9d640 10 monclient: get_auth_request con 0x7f20a806c870 auth_method 0 2023-12-01T17:36:36.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.481+0000 7f20bf7fe640 10 client.23230.objecter ms_handle_connect 0x7f20a806c870 2023-12-01T17:36:36.480 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.481+0000 7f20c6d9d640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f20a806c870 0x7f20a806ed30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f20b8036000 tx=0x7f20b8030000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.481 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f20c0057ea0 con 0x7f20c81423b0 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 10 client.23230.objecter ms_dispatch 0x7f20c8000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 3 client.23230.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 3 client.23230.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 10 client.23230.objecter checking command 1 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f20a8074460 0x7f20a8076900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 20 client.23230.objecter _get_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 20 client.23230.objecter put_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 15 client.23230.objecter _session_command_op_remove -1 1 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 20 client.23230.objecter _get_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 20 client.23230.objecter get_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 15 client.23230.objecter _session_command_op_assign 6 1 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 20 client.23230.objecter put_session s=0x7f20a8073d90 osd=6 4 2023-12-01T17:36:36.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 10 client.23230.objecter _send_command 1 2023-12-01T17:36:36.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 --> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f20a8076fd0 con 0x7f20a8074460 2023-12-01T17:36:36.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.482+0000 7f20bf7fe640 20 client.23230.objecter dump_active .. 0 homeless 2023-12-01T17:36:36.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.483+0000 7f20c659c640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f20a8074460 0x7f20a8076900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.483+0000 7f20c659c640 10 monclient: get_auth_request con 0x7f20a8074460 auth_method 0 2023-12-01T17:36:36.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.484+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=1099) v2 ==== 24+0+0 (secure 0 0 0) 0x7f20c00541e0 con 0x7f20c81423b0 2023-12-01T17:36:36.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.484+0000 7f20bf7fe640 10 monclient: handle_get_version_reply finishing 1 version 1099 2023-12-01T17:36:36.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.486+0000 7f20c659c640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f20a8074460 0x7f20a8076900 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.485 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.486+0000 7f20bf7fe640 10 client.23230.objecter ms_handle_connect 0x7f20a8074460 2023-12-01T17:36:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.510+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== osd.6 v2:172.21.15.92:6824/162801920 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f20a8076fd0 con 0x7f20a8074460 2023-12-01T17:36:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.510+0000 7f20bf7fe640 10 client.23230.objecter ms_dispatch 0x7f20c8000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.510+0000 7f20bf7fe640 10 client.23230.objecter _finish_command 1 = system:0 2023-12-01T17:36:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.510+0000 7f20bf7fe640 20 client.23230.objecter put_session s=0x7f20a8073d90 osd=6 4 2023-12-01T17:36:36.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.510+0000 7f20bf7fe640 15 client.23230.objecter _session_command_op_remove 6 1 2023-12-01T17:36:36.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 10 client.23230.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:36.526 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 15 client.23230.objecter _session_command_op_assign -1 2 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 20 client.23230.objecter _get_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 20 client.23230.objecter put_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 20 client.23230.objecter _get_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 15 client.23230.objecter _session_command_op_remove -1 2 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 20 client.23230.objecter get_session s=0x7f20a8073d90 osd=6 3 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 15 client.23230.objecter _session_command_op_assign 6 2 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 20 client.23230.objecter put_session s=0x7f20a8073d90 osd=6 4 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 10 client.23230.objecter _send_command 2 2023-12-01T17:36:36.527 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.527+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 --> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f20c80aacb0 con 0x7f20a8074460 2023-12-01T17:36:36.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20bf7fe640 1 -- 172.21.15.67:0/3253050648 <== osd.6 v2:172.21.15.92:6824/162801920 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f20b0009930 con 0x7f20a8074460 2023-12-01T17:36:36.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20bf7fe640 10 client.23230.objecter ms_dispatch 0x7f20c8000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:36.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20bf7fe640 10 client.23230.objecter _finish_command 2 = system:0 2023-12-01T17:36:36.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20bf7fe640 20 client.23230.objecter put_session s=0x7f20a8073d90 osd=6 4 2023-12-01T17:36:36.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20bf7fe640 15 client.23230.objecter _session_command_op_remove 6 2 2023-12-01T17:36:36.541 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:36.541 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:36.542 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:36.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 10 librados: watch_flush enter 2023-12-01T17:36:36.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 10 librados: watch_flush exit 2023-12-01T17:36:36.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 10 client.23230.objecter close_session for osd.6 2023-12-01T17:36:36.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f20a8074460 msgr2=0x7f20a8076900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:6824/162801920,v1:172.21.15.92:6825/162801920] conn(0x7f20a8074460 0x7f20a8076900 crc :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.545 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 20 client.23230.objecter put_session s=0x7f20a8073d90 osd=6 1 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 20 client.23230.objecter shutdown clearing up homeless session... 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 10 client.23230.objecter successfully canceled tick 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f20a806c870 msgr2=0x7f20a806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f20a806c870 0x7f20a806ed30 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f20b8036000 tx=0x7f20b8030000 comp rx=0 tx=0).stop 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 10 monclient: shutdown 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c81423b0 msgr2=0x7f20c8137b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 --2- 172.21.15.67:0/3253050648 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f20c81423b0 0x7f20c8137b40 secure :-1 s=READY pgs=5908 cs=0 l=1 rev1=1 crypto rx=0x7f20c000c3c0 tx=0x7f20c000c880 comp rx=0 tx=0).stop 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20c759e640 1 -- 172.21.15.67:0/3253050648 reap_dead start 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 shutdown_connections 2023-12-01T17:36:36.546 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 >> 172.21.15.67:0/3253050648 conn(0x7f20c80b7870 msgr2=0x7f20c813d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:36.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 shutdown_connections 2023-12-01T17:36:36.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 -- 172.21.15.67:0/3253050648 wait complete. 2023-12-01T17:36:36.547 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.541+0000 7f20cd061640 1 librados: shutdown 2023-12-01T17:36:36.553 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-12-01T17:36:36.661 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.662+0000 7fc5e8690640 10 monclient: get_monmap_and_config 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.662+0000 7fc5e8690640 10 monclient: build_initial_monmap 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.662+0000 7fc5e8690640 10 monclient: monmap: 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:36.663237+0000 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:36.663237+0000 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.662 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:36.663 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:36.663 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:36.663 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:36.663 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:36.663 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 1 Processor -- start 2023-12-01T17:36:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 1 -- start start 2023-12-01T17:36:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient: init 2023-12-01T17:36:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-12-01T17:36:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e006fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.665 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e00af380 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fc5e0072090 0x7fc5e00724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e0072090 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5e013b4c0 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e00702b0 con 0x7fc5e013b4c0 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e0071a70 con 0x7fc5e0072090 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00af760 con 0x7fc5e00af380 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.663+0000 7fc5e8690640 10 monclient(hunting): authenticate will time out at 3327.901293s 2023-12-01T17:36:36.666 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e642c640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e006fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e6c2d640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fc5e013b4c0 0x7fc5e013b910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:56742/0 (socket says 172.21.15.67:56742) 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e6c2d640 1 -- 172.21.15.67:0/3499078125 learned_addr learned my addr 172.21.15.67:0/3499078125 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e5c2b640 1 --2- 172.21.15.67:0/3499078125 >> v2:172.21.15.92:3300/0 conn(0x7fc5e0072090 0x7fc5e00724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e642c640 10 monclient(hunting): get_auth_request con 0x7fc5e00af380 auth_method 0 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e642c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e642c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e642c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e5c2b640 10 monclient(hunting): get_auth_request con 0x7fc5e0072090 auth_method 0 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e5c2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e5c2b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.664+0000 7fc5e5c2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.665+0000 7fc5e5c2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.665+0000 7fc5e5c2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.665+0000 7fc5e5c2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.665+0000 7fc5e642c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.665+0000 7fc5e642c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.665+0000 7fc5e642c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 10 monclient(hunting): handle_auth_done global_id 23233 payload 274 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 1 -- 172.21.15.67:0/3499078125 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 msgr2=0x7fc5e006fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 1 --2- 172.21.15.67:0/3499078125 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e006fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 1 -- 172.21.15.67:0/3499078125 >> v1:172.21.15.67:6789/0 conn(0x7fc5e013b4c0 legacy=0x7fc5e013b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 10 monclient: _finish_hunting 0 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 1 monclient: found mon.noname-c 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.667+0000 7fc5e5c2b640 1 -- 172.21.15.67:0/3499078125 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0070590 con 0x7fc5e0072090 2023-12-01T17:36:36.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.669+0000 7fc5e5c2b640 1 --2- 172.21.15.67:0/3499078125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e0072090 0x7fc5e00724b0 secure :-1 s=READY pgs=5909 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00097a0 tx=0x7fc5d002b7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98197a890115b9c server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 1 -- 172.21.15.67:0/3499078125 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d002f050 con 0x7fc5e0072090 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: dump: 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.672 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: _finish_auth 0 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: _check_auth_tickets 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 1 -- 172.21.15.67:0/3499078125 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d0031700 con 0x7fc5e0072090 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 1 -- 172.21.15.67:0/3499078125 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0031a90 con 0x7fc5e0072090 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e542a640 10 monclient: dump: 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.673 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e8690640 5 monclient: authenticate success, global_id 23233 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e8690640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:36.674 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.672+0000 7fc5e8690640 4 set_mon_vals no callback set 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 10 monclient: shutdown 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 -- 172.21.15.67:0/3499078125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e0072090 msgr2=0x7fc5e00724b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 --2- 172.21.15.67:0/3499078125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e0072090 0x7fc5e00724b0 secure :-1 s=READY pgs=5909 cs=0 l=1 rev1=1 crypto rx=0x7fc5d00097a0 tx=0x7fc5d002b7a0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 -- 172.21.15.67:0/3499078125 shutdown_connections 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 --2- 172.21.15.67:0/3499078125 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e0072090 0x7fc5e00724b0 unknown :-1 s=CLOSED pgs=5909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 --2- 172.21.15.67:0/3499078125 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e006fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 -- 172.21.15.67:0/3499078125 >> 172.21.15.67:0/3499078125 conn(0x7fc5e00b3ef0 msgr2=0x7fc5e00b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:36.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 -- 172.21.15.67:0/3499078125 shutdown_connections 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.673+0000 7fc5e8690640 1 -- 172.21.15.67:0/3499078125 wait complete. 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 10 monclient: build_initial_monmap 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 10 monclient: monmap: 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.679 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 1 librados: starting msgr at 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 1 librados: starting objecter 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 1 Processor -- start 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 1 -- start start 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 1 librados: setting wanted keys 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 1 librados: calling monclient init 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.674+0000 7fc5e8690640 10 monclient: init 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e0072090 0x7fc5e01351c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): picked mon.noname-a con 0x7fc5e0072090 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e012e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.680 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): picked mon.noname-c con 0x7fc5e00af380 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e013b4c0 0x7fc5e012eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): picked mon.noname-b con 0x7fc5e013b4c0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e01323c0 con 0x7fc5e0072090 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e01339f0 con 0x7fc5e013b4c0 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e012f1e0 con 0x7fc5e00af380 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e8690640 10 monclient(hunting): authenticate will time out at 3327.912598s 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.675+0000 7fc5e642c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e0072090 0x7fc5e01351c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e0072090 0x7fc5e01351c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37956/0 (socket says 172.21.15.67:37956) 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 1 -- 172.21.15.67:0/659170644 learned_addr learned my addr 172.21.15.67:0/659170644 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 10 monclient(hunting): get_auth_request con 0x7fc5e0072090 auth_method 0 2023-12-01T17:36:36.681 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e642c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e6c2d640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e013b4c0 0x7fc5e012eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e6c2d640 10 monclient(hunting): get_auth_request con 0x7fc5e013b4c0 auth_method 0 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e6c2d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e6c2d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e6c2d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.676+0000 7fc5e5c2b640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e012e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.677+0000 7fc5e6c2d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.677+0000 7fc5e6c2d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.677+0000 7fc5e6c2d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.677+0000 7fc5e5c2b640 10 monclient(hunting): get_auth_request con 0x7fc5e00af380 auth_method 0 2023-12-01T17:36:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.677+0000 7fc5e5c2b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.677+0000 7fc5e5c2b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.677+0000 7fc5e5c2b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.678+0000 7fc5e5c2b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.678+0000 7fc5e5c2b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:36.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.678+0000 7fc5e5c2b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 10 monclient(hunting): handle_auth_done global_id 23236 payload 824 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 1 -- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 msgr2=0x7fc5e012e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e012e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 1 -- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e0072090 msgr2=0x7fc5e01351c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e0072090 0x7fc5e01351c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 10 monclient: _finish_hunting 0 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 1 monclient: found mon.noname-b 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:36.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.680+0000 7fc5e6c2d640 1 -- 172.21.15.67:0/659170644 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d0009400 con 0x7fc5e013b4c0 2023-12-01T17:36:36.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.682+0000 7fc5e6c2d640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e013b4c0 0x7fc5e012eca0 secure :-1 s=READY pgs=5910 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800caa0 tx=0x7fc5d800cf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.682+0000 7fc5d6ffd640 10 client.?.objecter ms_handle_connect 0x7fc5e013b4c0 2023-12-01T17:36:36.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.682+0000 7fc5d6ffd640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:36.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 1 -- 172.21.15.67:0/659170644 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d800ab40 con 0x7fc5e013b4c0 2023-12-01T17:36:36.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 10 monclient: dump: 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 10 monclient: _finish_auth 0 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 10 monclient: _check_auth_tickets 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5e8690640 5 monclient: authenticate success, global_id 23236 2023-12-01T17:36:36.687 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 1 -- 172.21.15.67:0/659170644 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d800aca0 con 0x7fc5e013b4c0 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5d6ffd640 1 -- 172.21.15.67:0/659170644 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8012600 con 0x7fc5e013b4c0 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 10 monclient: _renew_subs 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e0131e10 con 0x7fc5e013b4c0 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.685+0000 7fc5e4c29640 4 set_mon_vals no callback set 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5d6ffd640 10 monclient: dump: 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:36.688 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 10 client.23236.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 10 monclient: _renew_subs 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e0131620 con 0x7fc5e013b4c0 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.686+0000 7fc5e8690640 1 librados: init done 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.687+0000 7fc5e8690640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.687+0000 7fc5e8690640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.687+0000 7fc5e8690640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:36.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.687+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e004e840 con 0x7fc5e013b4c0 2023-12-01T17:36:36.926 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.926+0000 7fc5d6ffd640 1 -- 172.21.15.67:0/659170644 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fc5d8007070 con 0x7fc5e013b4c0 2023-12-01T17:36:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.927+0000 7fc5d6ffd640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc5c406c870 0x7fc5c406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.927+0000 7fc5e642c640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc5c406c870 0x7fc5c406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.927+0000 7fc5e642c640 10 monclient: get_auth_request con 0x7fc5c406c870 auth_method 0 2023-12-01T17:36:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.927+0000 7fc5d6ffd640 10 client.23236.objecter ms_handle_connect 0x7fc5c406c870 2023-12-01T17:36:36.927 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.927+0000 7fc5e642c640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc5c406c870 0x7fc5c406ed30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009c00 tx=0x7fc5dc009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:36.928 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.929+0000 7fc5d6ffd640 1 -- 172.21.15.67:0/659170644 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fc5d80586b0 con 0x7fc5e013b4c0 2023-12-01T17:36:36.928 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.929+0000 7fc5d6ffd640 10 client.23236.objecter ms_dispatch 0x7fc5e0000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:36.928 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.929+0000 7fc5d6ffd640 3 client.23236.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.929+0000 7fc5d6ffd640 3 client.23236.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:36.929 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.929+0000 7fc5d6ffd640 20 client.23236.objecter dump_active .. 0 homeless 2023-12-01T17:36:36.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.939+0000 7fc5d6ffd640 1 -- 172.21.15.67:0/659170644 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5d80549f0 con 0x7fc5e013b4c0 2023-12-01T17:36:36.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.939+0000 7fc5d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:36.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:36.939+0000 7fc5d6ffd640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:37.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.120+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5e0070ab0 con 0x7fc5c406c870 2023-12-01T17:36:37.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.120+0000 7fc5d6ffd640 1 -- 172.21.15.67:0/659170644 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc5dc011b20 con 0x7fc5c406c870 2023-12-01T17:36:37.120 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 6 to deep-scrub 2023-12-01T17:36:37.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 10 librados: watch_flush enter 2023-12-01T17:36:37.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 10 librados: watch_flush exit 2023-12-01T17:36:37.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 20 client.23236.objecter shutdown clearing up homeless session... 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 10 client.23236.objecter successfully canceled tick 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc5c406c870 msgr2=0x7fc5c406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc5c406c870 0x7fc5c406ed30 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009c00 tx=0x7fc5dc009190 comp rx=0 tx=0).stop 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 10 monclient: shutdown 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e013b4c0 msgr2=0x7fc5e012eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e013b4c0 0x7fc5e012eca0 secure :-1 s=READY pgs=5910 cs=0 l=1 rev1=1 crypto rx=0x7fc5d800caa0 tx=0x7fc5d800cf60 comp rx=0 tx=0).stop 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 shutdown_connections 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc5e013b4c0 0x7fc5e012eca0 unknown :-1 s=CLOSED pgs=5910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fc5c406c870 0x7fc5c406ed30 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fc5e00af380 0x7fc5e012e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 --2- 172.21.15.67:0/659170644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5e0072090 0x7fc5e01351c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 >> 172.21.15.67:0/659170644 conn(0x7fc5e00b3ef0 msgr2=0x7fc5e0073cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 shutdown_connections 2023-12-01T17:36:37.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 -- 172.21.15.67:0/659170644 wait complete. 2023-12-01T17:36:37.124 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.122+0000 7fc5e8690640 1 librados: shutdown 2023-12-01T17:36:37.133 INFO:tasks.ceph:Scrubbing osd.7 2023-12-01T17:36:37.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-12-01T17:36:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.245+0000 7fda4791d640 10 monclient: get_monmap_and_config 2023-12-01T17:36:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.245+0000 7fda4791d640 10 monclient: build_initial_monmap 2023-12-01T17:36:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.245+0000 7fda4791d640 10 monclient: monmap: 2023-12-01T17:36:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.245 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:37.246682+0000 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:37.246682+0000 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:37.246 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.246+0000 7fda4791d640 1 Processor -- start 2023-12-01T17:36:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.246+0000 7fda4791d640 1 -- start start 2023-12-01T17:36:37.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.246+0000 7fda4791d640 10 monclient: init 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda400a8a20 0x7fda400a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient(hunting): picked mon.noname-a con 0x7fda400a8a20 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda40145ab0 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7fda4013f800 0x7fda40145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient(hunting): picked mon.noname-c con 0x7fda4013f800 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda40146430 con 0x7fda40145ab0 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda40145620 con 0x7fda400a8a20 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda400aace0 con 0x7fda4013f800 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:37.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.247+0000 7fda4791d640 10 monclient(hunting): authenticate will time out at 3328.484787s 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.248+0000 7fda456b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda400a8a20 0x7fda400a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.248+0000 7fda456b9640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fda400a8a20 0x7fda400a8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37962/0 (socket says 172.21.15.67:37962) 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.248+0000 7fda456b9640 1 -- 172.21.15.67:0/993519557 learned_addr learned my addr 172.21.15.67:0/993519557 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.248+0000 7fda456b9640 10 monclient(hunting): get_auth_request con 0x7fda400a8a20 auth_method 0 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.248+0000 7fda456b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.248+0000 7fda456b9640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.248+0000 7fda456b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.249+0000 7fda456b9640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:37.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.249+0000 7fda456b9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.249+0000 7fda456b9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 10 monclient(hunting): handle_auth_done global_id 16044 payload 274 2023-12-01T17:36:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 1 -- 172.21.15.67:0/993519557 >> v2:172.21.15.92:3300/0 conn(0x7fda4013f800 msgr2=0x7fda40145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 1 --2- 172.21.15.67:0/993519557 >> v2:172.21.15.92:3300/0 conn(0x7fda4013f800 0x7fda40145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 1 -- 172.21.15.67:0/993519557 >> v1:172.21.15.92:6789/0 conn(0x7fda40145ab0 legacy=0x7fda4013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 10 monclient: _finish_hunting 0 2023-12-01T17:36:37.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 1 monclient: found mon.noname-a 2023-12-01T17:36:37.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:37.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.250+0000 7fda456b9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:36:37.251 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.251+0000 7fda456b9640 1 -- 172.21.15.67:0/993519557 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda400aafc0 con 0x7fda400a8a20 2023-12-01T17:36:37.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.253+0000 7fda456b9640 1 --2- 172.21.15.67:0/993519557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 0x7fda400a8e00 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7fda400b87f0 tx=0x7fda3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906c00d783a8e47a server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 1 -- 172.21.15.67:0/993519557 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30031870 con 0x7fda400a8a20 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: dump: 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.256 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: _finish_auth 0 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 1 -- 172.21.15.67:0/993519557 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda300319d0 con 0x7fda400a8a20 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda4791d640 5 monclient: authenticate success, global_id 16044 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda4791d640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda4791d640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda4791d640 4 set_mon_vals no callback set 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 1 -- 172.21.15.67:0/993519557 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30031d60 con 0x7fda400a8a20 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.256+0000 7fda37fff640 10 monclient: dump: 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.258 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 10 monclient: shutdown 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 -- 172.21.15.67:0/993519557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 msgr2=0x7fda400a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 --2- 172.21.15.67:0/993519557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 0x7fda400a8e00 secure :-1 s=READY pgs=3176 cs=0 l=1 rev1=1 crypto rx=0x7fda400b87f0 tx=0x7fda3002b910 comp rx=0 tx=0).stop 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 -- 172.21.15.67:0/993519557 shutdown_connections 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 --2- 172.21.15.67:0/993519557 >> v2:172.21.15.92:3300/0 conn(0x7fda4013f800 0x7fda40145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 --2- 172.21.15.67:0/993519557 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 0x7fda400a8e00 unknown :-1 s=CLOSED pgs=3176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 -- 172.21.15.67:0/993519557 >> 172.21.15.67:0/993519557 conn(0x7fda400b7890 msgr2=0x7fda400b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 -- 172.21.15.67:0/993519557 shutdown_connections 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.257+0000 7fda4791d640 1 -- 172.21.15.67:0/993519557 wait complete. 2023-12-01T17:36:37.260 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 10 monclient: build_initial_monmap 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 10 monclient: monmap: 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 1 librados: starting msgr at 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 1 librados: starting objecter 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 1 Processor -- start 2023-12-01T17:36:37.261 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 1 -- start start 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 1 librados: setting wanted keys 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 1 librados: calling monclient init 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.258+0000 7fda4791d640 10 monclient: init 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient: _add_conns ranks=[0,2,1] 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 0x7fda40145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): picked mon.noname-a con 0x7fda400a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fda4013f800 0x7fda400b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): picked mon.noname-c con 0x7fda4013f800 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40145ab0 0x7fda401370d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): picked mon.noname-b con 0x7fda40145ab0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda401329e0 con 0x7fda400a8a20 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.262 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda401da880 con 0x7fda40145ab0 2023-12-01T17:36:37.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda40138f70 con 0x7fda4013f800 2023-12-01T17:36:37.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:37.263 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.259+0000 7fda4791d640 10 monclient(hunting): authenticate will time out at 3328.496623s 2023-12-01T17:36:37.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.261+0000 7fda45eba640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40145ab0 0x7fda401370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40145ab0 0x7fda401370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42800/0 (socket says 172.21.15.67:42800) 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 1 -- 172.21.15.67:0/2616593641 learned_addr learned my addr 172.21.15.67:0/2616593641 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 10 monclient(hunting): get_auth_request con 0x7fda40145ab0 auth_method 0 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.262+0000 7fda45eba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.263+0000 7fda456b9640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 0x7fda40145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.263+0000 7fda44eb8640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fda4013f800 0x7fda400b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda456b9640 10 monclient(hunting): get_auth_request con 0x7fda400a8a20 auth_method 0 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda456b9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda456b9640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda456b9640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda44eb8640 10 monclient(hunting): get_auth_request con 0x7fda4013f800 auth_method 0 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda44eb8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda44eb8640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.264+0000 7fda44eb8640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 10 monclient(hunting): handle_auth_done global_id 23239 payload 824 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 1 -- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fda4013f800 msgr2=0x7fda400b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7fda4013f800 0x7fda400b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 1 -- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 msgr2=0x7fda40145040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fda400a8a20 0x7fda40145040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 10 monclient: _finish_hunting 0 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 1 monclient: found mon.noname-b 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.265+0000 7fda45eba640 1 -- 172.21.15.67:0/2616593641 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda300094d0 con 0x7fda40145ab0 2023-12-01T17:36:37.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.267+0000 7fda45eba640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40145ab0 0x7fda401370d0 secure :-1 s=READY pgs=5911 cs=0 l=1 rev1=1 crypto rx=0x7fda3800b970 tx=0x7fda3800be30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:37.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.267+0000 7fda35ffb640 10 client.?.objecter ms_handle_connect 0x7fda40145ab0 2023-12-01T17:36:37.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.267+0000 7fda35ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3800cb40 con 0x7fda40145ab0 2023-12-01T17:36:37.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 10 monclient: dump: 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:37.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda3800ccc0 con 0x7fda40145ab0 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda38014680 con 0x7fda40145ab0 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda4791d640 5 monclient: authenticate success, global_id 23239 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 monclient: _renew_subs 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda40139250 con 0x7fda40145ab0 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.271+0000 7fda377fe640 4 set_mon_vals no callback set 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda35ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda35ffb640 10 monclient: dump: 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.273 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 client.23239.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 monclient: _renew_subs 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda401db190 con 0x7fda40145ab0 2023-12-01T17:36:37.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 1 librados: init done 2023-12-01T17:36:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 client.23239.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 15 client.23239.objecter _session_command_op_assign -1 1 2023-12-01T17:36:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 20 client.23239.objecter _get_session osd=-1 returning homeless 2023-12-01T17:36:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 client.23239.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.272+0000 7fda4791d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.273+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda4014c860 con 0x7fda40145ab0 2023-12-01T17:36:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.293+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7fda3801d3b0 con 0x7fda40145ab0 2023-12-01T17:36:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.293+0000 7fda35ffb640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fda2406c820 0x7fda2406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.294+0000 7fda456b9640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fda2406c820 0x7fda2406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.294+0000 7fda456b9640 10 monclient: get_auth_request con 0x7fda2406c820 auth_method 0 2023-12-01T17:36:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.294+0000 7fda456b9640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fda2406c820 0x7fda2406ece0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fda30036000 tx=0x7fda3003c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:37.293 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.294+0000 7fda35ffb640 10 client.23239.objecter ms_handle_connect 0x7fda2406c820 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7fda3800ce30 con 0x7fda40145ab0 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 10 client.23239.objecter ms_dispatch 0x7fda40000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 3 client.23239.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 3 client.23239.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 10 client.23239.objecter checking command 1 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fda24074410 0x7fda240768b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 20 client.23239.objecter _get_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 20 client.23239.objecter put_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 15 client.23239.objecter _session_command_op_remove -1 1 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 20 client.23239.objecter _get_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 20 client.23239.objecter get_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 15 client.23239.objecter _session_command_op_assign 7 1 2023-12-01T17:36:37.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 20 client.23239.objecter put_session s=0x7fda24073d40 osd=7 4 2023-12-01T17:36:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 10 client.23239.objecter _send_command 1 2023-12-01T17:36:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 --> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda24076f80 con 0x7fda24074410 2023-12-01T17:36:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda35ffb640 20 client.23239.objecter dump_active .. 0 homeless 2023-12-01T17:36:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.296+0000 7fda44eb8640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fda24074410 0x7fda240768b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.297+0000 7fda44eb8640 10 monclient: get_auth_request con 0x7fda24074410 auth_method 0 2023-12-01T17:36:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.302+0000 7fda44eb8640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fda24074410 0x7fda240768b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:37.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.302+0000 7fda35ffb640 10 client.23239.objecter ms_handle_connect 0x7fda24074410 2023-12-01T17:36:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.330+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== osd.7 v2:172.21.15.92:6808/2695530702 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fda24076f80 con 0x7fda24074410 2023-12-01T17:36:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.330+0000 7fda35ffb640 10 client.23239.objecter ms_dispatch 0x7fda40000b90 command_reply(tid 1: 0 ) v1 2023-12-01T17:36:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.330+0000 7fda35ffb640 10 client.23239.objecter _finish_command 1 = system:0 2023-12-01T17:36:37.329 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.330+0000 7fda35ffb640 20 client.23239.objecter put_session s=0x7fda24073d40 osd=7 4 2023-12-01T17:36:37.330 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.330+0000 7fda35ffb640 15 client.23239.objecter _session_command_op_remove 7 1 2023-12-01T17:36:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.346+0000 7fda4791d640 10 client.23239.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-12-01T17:36:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.346+0000 7fda4791d640 15 client.23239.objecter _session_command_op_assign -1 2 2023-12-01T17:36:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.346+0000 7fda4791d640 20 client.23239.objecter _get_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.346 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 20 client.23239.objecter put_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 20 client.23239.objecter _get_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 15 client.23239.objecter _session_command_op_remove -1 2 2023-12-01T17:36:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 20 client.23239.objecter get_session s=0x7fda24073d40 osd=7 3 2023-12-01T17:36:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 15 client.23239.objecter _session_command_op_assign 7 2 2023-12-01T17:36:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 20 client.23239.objecter put_session s=0x7fda24073d40 osd=7 4 2023-12-01T17:36:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 10 client.23239.objecter _send_command 2 2023-12-01T17:36:37.347 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.347+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 --> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fda4004e840 con 0x7fda24074410 2023-12-01T17:36:37.360 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== osd.7 v2:172.21.15.92:6808/2695530702 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fda4004e840 con 0x7fda24074410 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda35ffb640 10 client.23239.objecter ms_dispatch 0x7fda40000b90 command_reply(tid 2: 0 ) v1 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda35ffb640 10 client.23239.objecter _finish_command 2 = system:0 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda35ffb640 20 client.23239.objecter put_session s=0x7fda24073d40 osd=7 4 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda35ffb640 15 client.23239.objecter _session_command_op_remove 7 2 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda35ffb640 1 -- 172.21.15.67:0/2616593641 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_get_version_reply(handle=1 version=1099) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda38054d60 con 0x7fda40145ab0 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda35ffb640 10 monclient: handle_get_version_reply finishing 1 version 1099 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-01T17:36:37.361 INFO:teuthology.orchestra.run.smithi067.stdout:} 2023-12-01T17:36:37.362 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 10 librados: watch_flush enter 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 10 librados: watch_flush exit 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 10 client.23239.objecter close_session for osd.7 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fda24074410 msgr2=0x7fda240768b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:6808/2695530702,v1:172.21.15.92:6809/2695530702] conn(0x7fda24074410 0x7fda240768b0 crc :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 20 client.23239.objecter put_session s=0x7fda24073d40 osd=7 1 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 20 client.23239.objecter shutdown clearing up homeless session... 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 10 client.23239.objecter successfully canceled tick 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fda2406c820 msgr2=0x7fda2406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7fda2406c820 0x7fda2406ece0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fda30036000 tx=0x7fda3003c040 comp rx=0 tx=0).stop 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 10 monclient: shutdown 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40145ab0 msgr2=0x7fda401370d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 --2- 172.21.15.67:0/2616593641 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fda40145ab0 0x7fda401370d0 secure :-1 s=READY pgs=5911 cs=0 l=1 rev1=1 crypto rx=0x7fda3800b970 tx=0x7fda3800be30 comp rx=0 tx=0).stop 2023-12-01T17:36:37.363 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda45eba640 1 -- 172.21.15.67:0/2616593641 reap_dead start 2023-12-01T17:36:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 shutdown_connections 2023-12-01T17:36:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 >> 172.21.15.67:0/2616593641 conn(0x7fda400b7890 msgr2=0x7fda40135d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 shutdown_connections 2023-12-01T17:36:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 -- 172.21.15.67:0/2616593641 wait complete. 2023-12-01T17:36:37.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.361+0000 7fda4791d640 1 librados: shutdown 2023-12-01T17:36:37.374 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-12-01T17:36:37.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.482+0000 7f8ae6ede640 10 monclient: get_monmap_and_config 2023-12-01T17:36:37.482 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.482+0000 7f8ae6ede640 10 monclient: build_initial_monmap 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.483+0000 7f8ae6ede640 10 monclient: monmap: 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:37.483753+0000 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:37.483753+0000 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.483 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:37.484 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:37.484 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:37.484 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:37.484 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:37.484 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.483+0000 7f8ae6ede640 1 Processor -- start 2023-12-01T17:36:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.483+0000 7f8ae6ede640 1 -- start start 2023-12-01T17:36:37.488 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.483+0000 7f8ae6ede640 10 monclient: init 2023-12-01T17:36:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-12-01T17:36:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae00a8a20 0x7f8ae013eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae00a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ae00a8e00 0x7f8ae013f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae00a8e00 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:37.489 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8ae01423b0 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae00b21f0 con 0x7f8ae01423b0 2023-12-01T17:36:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae00a9cc0 con 0x7f8ae00a8e00 2023-12-01T17:36:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0142f50 con 0x7f8ae00a8a20 2023-12-01T17:36:37.490 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.484+0000 7f8ae6ede640 10 monclient(hunting): authenticate will time out at 3328.721874s 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8adffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ae00a8e00 0x7f8ae013f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8ae4c7a640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae00a8a20 0x7f8ae013eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8adffff640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8ae00a8e00 0x7f8ae013f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37978/0 (socket says 172.21.15.67:37978) 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8adffff640 1 -- 172.21.15.67:0/1454242185 learned_addr learned my addr 172.21.15.67:0/1454242185 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8adffff640 10 monclient(hunting): get_auth_request con 0x7f8ae00a8e00 auth_method 0 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8adffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8adffff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.491 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8adffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8ae4c7a640 10 monclient(hunting): get_auth_request con 0x7f8ae00a8a20 auth_method 0 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8ae4c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8ae4c7a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.485+0000 7f8ae4c7a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.486+0000 7f8ae4c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.486+0000 7f8ae4c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.486+0000 7f8ae4c7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.486+0000 7f8adffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.486+0000 7f8adffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.486+0000 7f8adffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 10 monclient(hunting): handle_auth_done global_id 17981 payload 274 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 1 -- 172.21.15.67:0/1454242185 >> v2:172.21.15.67:3300/0 conn(0x7f8ae00a8e00 msgr2=0x7f8ae013f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 1 --2- 172.21.15.67:0/1454242185 >> v2:172.21.15.67:3300/0 conn(0x7f8ae00a8e00 0x7f8ae013f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 1 -- 172.21.15.67:0/1454242185 >> v1:172.21.15.67:6789/0 conn(0x7f8ae01423b0 legacy=0x7f8ae0142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 10 monclient: _finish_hunting 0 2023-12-01T17:36:37.492 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 1 monclient: found mon.noname-b 2023-12-01T17:36:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.67:3301/0 2023-12-01T17:36:37.493 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.487+0000 7f8ae4c7a640 1 -- 172.21.15.67:0/1454242185 --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae013f930 con 0x7f8ae00a8a20 2023-12-01T17:36:37.494 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.490+0000 7f8ae4c7a640 1 --2- 172.21.15.67:0/1454242185 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae00a8a20 0x7f8ae013eeb0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80097a0 tx=0x7f8ad802b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80c305a80446ea2f server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 1 -- 172.21.15.67:0/1454242185 <== mon.2 v2:172.21.15.67:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad802f050 con 0x7f8ae00a8a20 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: dump: 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.495 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: _finish_auth 0 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: _check_auth_tickets 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 5 monclient: authenticate success, global_id 17981 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 1 -- 172.21.15.67:0/1454242185 <== mon.2 v2:172.21.15.67:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad8031700 con 0x7f8ae00a8a20 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 4 set_mon_vals no callback set 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 1 -- 172.21.15.67:0/1454242185 <== mon.2 v2:172.21.15.67:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8031a90 con 0x7f8ae00a8a20 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8adf7fe640 10 monclient: dump: 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.496 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 10 monclient: shutdown 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 -- 172.21.15.67:0/1454242185 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae00a8a20 msgr2=0x7f8ae013eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/1454242185 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae00a8a20 0x7f8ae013eeb0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80097a0 tx=0x7f8ad802b7a0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 -- 172.21.15.67:0/1454242185 shutdown_connections 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/1454242185 >> v2:172.21.15.67:3300/0 conn(0x7f8ae00a8e00 0x7f8ae013f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/1454242185 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae00a8a20 0x7f8ae013eeb0 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.497 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 -- 172.21.15.67:0/1454242185 >> 172.21.15.67:0/1454242185 conn(0x7f8ae00b7870 msgr2=0x7f8ae00b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 -- 172.21.15.67:0/1454242185 shutdown_connections 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.494+0000 7f8ae6ede640 1 -- 172.21.15.67:0/1454242185 wait complete. 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.495+0000 7f8ae6ede640 10 monclient: build_initial_monmap 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.495+0000 7f8ae6ede640 10 monclient: monmap: 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:37.498 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:37.499 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:37.499 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.495+0000 7f8ae6ede640 1 librados: starting msgr at 2023-12-01T17:36:37.499 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.495+0000 7f8ae6ede640 1 librados: starting objecter 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.495+0000 7f8ae6ede640 1 Processor -- start 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 -- start start 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 librados: setting wanted keys 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 librados: calling monclient init 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient: init 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae00a8a20 0x7f8ae01443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): picked mon.noname-a con 0x7f8ae00a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00a8e00 0x7f8ae00b2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): picked mon.noname-b con 0x7f8ae00a8e00 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae01423b0 0x7f8ae0145df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.502 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): picked mon.noname-c con 0x7f8ae01423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0139470 con 0x7f8ae00a8a20 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae013a7a0 con 0x7f8ae00a8e00 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae013aa80 con 0x7f8ae01423b0 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.496+0000 7f8ae6ede640 10 monclient(hunting): authenticate will time out at 3328.734119s 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae4c7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae00a8a20 0x7f8ae01443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae547b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae01423b0 0x7f8ae0145df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8adffff640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00a8e00 0x7f8ae00b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae4c7a640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae00a8a20 0x7f8ae01443e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37980/0 (socket says 172.21.15.67:37980) 2023-12-01T17:36:37.503 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae547b640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae01423b0 0x7f8ae0145df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3301/0 says I am v2:172.21.15.67:45914/0 (socket says 172.21.15.67:45914) 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae4c7a640 1 -- 172.21.15.67:0/505353322 learned_addr learned my addr 172.21.15.67:0/505353322 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae4c7a640 10 monclient(hunting): get_auth_request con 0x7f8ae00a8a20 auth_method 0 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae4c7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae4c7a640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae4c7a640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae547b640 10 monclient(hunting): get_auth_request con 0x7f8ae01423b0 auth_method 0 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae547b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae547b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8ae547b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8adffff640 10 monclient(hunting): get_auth_request con 0x7f8ae00a8e00 auth_method 0 2023-12-01T17:36:37.504 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8adffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8adffff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.497+0000 7f8adffff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8ae547b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8ae547b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8ae547b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:37.505 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8ae4c7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8ae4c7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8ae4c7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8adffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8adffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.498+0000 7f8adffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 10 monclient(hunting): handle_auth_done global_id 23242 payload 824 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 1 -- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae01423b0 msgr2=0x7f8ae0145df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae01423b0 0x7f8ae0145df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.506 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 1 -- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae00a8a20 msgr2=0x7f8ae01443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae00a8a20 0x7f8ae01443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 10 monclient: _finish_hunting 0 2023-12-01T17:36:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 1 monclient: found mon.noname-b 2023-12-01T17:36:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.507 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.500+0000 7f8adffff640 1 -- 172.21.15.67:0/505353322 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad8009400 con 0x7f8ae00a8e00 2023-12-01T17:36:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.503+0000 7f8adffff640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00a8e00 0x7f8ae00b2380 secure :-1 s=READY pgs=5912 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000ef90 tx=0x7f8ad000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.503+0000 7f8add7fa640 10 client.?.objecter ms_handle_connect 0x7f8ae00a8e00 2023-12-01T17:36:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.503+0000 7f8add7fa640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 1 -- 172.21.15.67:0/505353322 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad000cda0 con 0x7f8ae00a8e00 2023-12-01T17:36:37.508 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: dump: 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.509 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: _finish_auth 0 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: _check_auth_tickets 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 1 -- 172.21.15.67:0/505353322 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad000f040 con 0x7f8ae00a8e00 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 1 -- 172.21.15.67:0/505353322 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0014680 con 0x7f8ae00a8e00 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8add7fa640 10 monclient: dump: 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:37.510 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8ae6ede640 5 monclient: authenticate success, global_id 23242 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8ae6ede640 10 monclient: _renew_subs 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8adeffd640 4 set_mon_vals no callback set 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8ae6ede640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.506+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae01397d0 con 0x7f8ae00a8e00 2023-12-01T17:36:37.511 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 10 client.23242.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 10 monclient: _renew_subs 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae0146330 con 0x7f8ae00a8e00 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 1 librados: init done 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:37.512 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.507+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae004e840 con 0x7f8ae00a8e00 2023-12-01T17:36:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.744+0000 7f8add7fa640 1 -- 172.21.15.67:0/505353322 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8ad00147e0 con 0x7f8ae00a8e00 2023-12-01T17:36:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.744+0000 7f8add7fa640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8ac006c870 0x7f8ac006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.745+0000 7f8ae4c7a640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8ac006c870 0x7f8ac006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:37.744 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.745+0000 7f8ae4c7a640 10 monclient: get_auth_request con 0x7f8ac006c870 auth_method 0 2023-12-01T17:36:37.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.745+0000 7f8add7fa640 10 client.23242.objecter ms_handle_connect 0x7f8ac006c870 2023-12-01T17:36:37.745 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.745+0000 7f8ae4c7a640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8ac006c870 0x7f8ac006ed30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8ad802b7e0 tx=0x7f8ad8008030 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.747+0000 7f8add7fa640 1 -- 172.21.15.67:0/505353322 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f8ad0059500 con 0x7f8ae00a8e00 2023-12-01T17:36:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.747+0000 7f8add7fa640 10 client.23242.objecter ms_dispatch 0x7f8ae0000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.747+0000 7f8add7fa640 3 client.23242.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.747+0000 7f8add7fa640 3 client.23242.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:37.747 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.747+0000 7f8add7fa640 20 client.23242.objecter dump_active .. 0 homeless 2023-12-01T17:36:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.757+0000 7f8add7fa640 1 -- 172.21.15.67:0/505353322 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ad0055840 con 0x7f8ae00a8e00 2023-12-01T17:36:37.756 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.757+0000 7f8add7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:37.757 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.757+0000 7f8add7fa640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:37.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.936+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ae013abf0 con 0x7f8ac006c870 2023-12-01T17:36:37.936 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.937+0000 7f8add7fa640 1 -- 172.21.15.67:0/505353322 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8ae013abf0 con 0x7f8ac006c870 2023-12-01T17:36:37.936 INFO:teuthology.orchestra.run.smithi067.stderr:instructed osd(s) 7 to deep-scrub 2023-12-01T17:36:37.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.938+0000 7f8ae6ede640 10 librados: watch_flush enter 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.938+0000 7f8ae6ede640 10 librados: watch_flush exit 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.938+0000 7f8ae6ede640 20 client.23242.objecter shutdown clearing up homeless session... 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.938+0000 7f8ae6ede640 10 client.23242.objecter successfully canceled tick 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8ac006c870 msgr2=0x7f8ac006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8ac006c870 0x7f8ac006ed30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f8ad802b7e0 tx=0x7f8ad8008030 comp rx=0 tx=0).stop 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 10 monclient: shutdown 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00a8e00 msgr2=0x7f8ae00b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00a8e00 0x7f8ae00b2380 secure :-1 s=READY pgs=5912 cs=0 l=1 rev1=1 crypto rx=0x7f8ad000ef90 tx=0x7f8ad000c490 comp rx=0 tx=0).stop 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 shutdown_connections 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8ac006c870 0x7f8ac006ed30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8ae01423b0 0x7f8ae0145df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8ae00a8e00 0x7f8ae00b2380 unknown :-1 s=CLOSED pgs=5912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 --2- 172.21.15.67:0/505353322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ae00a8a20 0x7f8ae01443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 >> 172.21.15.67:0/505353322 conn(0x7f8ae00b7870 msgr2=0x7f8ae00b6bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 shutdown_connections 2023-12-01T17:36:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 -- 172.21.15.67:0/505353322 wait complete. 2023-12-01T17:36:37.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:37.939+0000 7f8ae6ede640 1 librados: shutdown 2023-12-01T17:36:37.949 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:36:38.068 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.069+0000 7febe9e46640 10 monclient: get_monmap_and_config 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.069+0000 7febe9e46640 10 monclient: build_initial_monmap 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.069+0000 7febe9e46640 10 monclient: monmap: 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:38.069917+0000 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:38.069917+0000 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:38.069 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.069+0000 7febe9e46640 1 Processor -- start 2023-12-01T17:36:38.071 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.069+0000 7febe9e46640 1 -- start start 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.069+0000 7febe9e46640 10 monclient: init 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7febe40a8a20 addr v1:172.21.15.92:6789/0 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7febe41461c0 0x7febe40a8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient(hunting): picked mon.noname-a con 0x7febe41461c0 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7febe413f0f0 0x7febe413f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient(hunting): picked mon.noname-c con 0x7febe413f0f0 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe4145620 con 0x7febe40a8a20 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe40aace0 con 0x7febe41461c0 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe40aafc0 con 0x7febe413f0f0 2023-12-01T17:36:38.072 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.070+0000 7febe9e46640 10 monclient(hunting): authenticate will time out at 3329.307973s 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe3fff640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7febe413f0f0 0x7febe413f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe2ffd640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7febe41461c0 0x7febe40a8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe37fe640 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7febe40a8a20 0x7febe4145ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.67:36998/0 (socket says 172.21.15.67:36998) 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe37fe640 1 -- 172.21.15.67:0/3413351173 learned_addr learned my addr 172.21.15.67:0/3413351173 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe2ffd640 10 monclient(hunting): get_auth_request con 0x7febe41461c0 auth_method 0 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe3fff640 10 monclient(hunting): get_auth_request con 0x7febe413f0f0 auth_method 0 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe3fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.071+0000 7febe3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.072+0000 7febe2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:38.073 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.072+0000 7febe2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:38.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.072+0000 7febe2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:38.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.072+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:38.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.072+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:38.074 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.072+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.073+0000 7febe2ffd640 10 monclient(hunting): handle_auth_done global_id 16059 payload 274 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.073+0000 7febe2ffd640 1 -- 172.21.15.67:0/3413351173 >> v2:172.21.15.92:3300/0 conn(0x7febe413f0f0 msgr2=0x7febe413f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.073+0000 7febe2ffd640 1 --2- 172.21.15.67:0/3413351173 >> v2:172.21.15.92:3300/0 conn(0x7febe413f0f0 0x7febe413f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.073+0000 7febe2ffd640 1 -- 172.21.15.67:0/3413351173 >> v1:172.21.15.92:6789/0 conn(0x7febe40a8a20 legacy=0x7febe4145ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.073+0000 7febe2ffd640 10 monclient: _finish_hunting 0 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.073+0000 7febe2ffd640 1 monclient: found mon.noname-a 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.073+0000 7febe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.074+0000 7febe2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.67:3300/0 2023-12-01T17:36:38.075 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.074+0000 7febe2ffd640 1 -- 172.21.15.67:0/3413351173 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe413fa90 con 0x7febe41461c0 2023-12-01T17:36:38.076 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.075+0000 7febe2ffd640 1 --2- 172.21.15.67:0/3413351173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe41461c0 0x7febe40a8e00 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7febe40b3990 tx=0x7febd802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46884a57a17192bd server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:38.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 1 -- 172.21.15.67:0/3413351173 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd8031830 con 0x7febe41461c0 2023-12-01T17:36:38.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:38.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-12-01T17:36:38.079 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: dump: 2023-12-01T17:36:38.079 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:38.079 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: _finish_auth 0 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: _check_auth_tickets 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 1 -- 172.21.15.67:0/3413351173 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8031990 con 0x7febe41461c0 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:38.080 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 1 -- 172.21.15.67:0/3413351173 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd8031d20 con 0x7febe41461c0 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe9e46640 5 monclient: authenticate success, global_id 16059 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe9e46640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe9e46640 4 set_mon_vals no callback set 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.079+0000 7febe27fc640 10 monclient: dump: 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:38.081 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:38.082 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 10 monclient: shutdown 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 -- 172.21.15.67:0/3413351173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe41461c0 msgr2=0x7febe40a8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 --2- 172.21.15.67:0/3413351173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe41461c0 0x7febe40a8e00 secure :-1 s=READY pgs=3178 cs=0 l=1 rev1=1 crypto rx=0x7febe40b3990 tx=0x7febd802b8d0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 -- 172.21.15.67:0/3413351173 shutdown_connections 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 --2- 172.21.15.67:0/3413351173 >> v2:172.21.15.92:3300/0 conn(0x7febe413f0f0 0x7febe413f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 --2- 172.21.15.67:0/3413351173 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe41461c0 0x7febe40a8e00 unknown :-1 s=CLOSED pgs=3178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 -- 172.21.15.67:0/3413351173 >> 172.21.15.67:0/3413351173 conn(0x7febe40b7890 msgr2=0x7febe40b7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 -- 172.21.15.67:0/3413351173 shutdown_connections 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.080+0000 7febe9e46640 1 -- 172.21.15.67:0/3413351173 wait complete. 2023-12-01T17:36:38.085 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.081+0000 7febe9e46640 10 monclient: build_initial_monmap 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.081+0000 7febe9e46640 10 monclient: monmap: 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.081+0000 7febe9e46640 1 librados: starting msgr at 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.081+0000 7febe9e46640 1 librados: starting objecter 2023-12-01T17:36:38.086 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.081+0000 7febe9e46640 1 Processor -- start 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 -- start start 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 librados: setting wanted keys 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 librados: calling monclient init 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient: init 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient: _add_conns ranks=[0,1,2] 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe40a8a20 0x7febe4144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): picked mon.noname-a con 0x7febe40a8a20 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febe413f0f0 0x7febe4136ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): picked mon.noname-b con 0x7febe413f0f0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7febe41461c0 0x7febe41db080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): picked mon.noname-c con 0x7febe41461c0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4138e40 con 0x7febe40a8a20 2023-12-01T17:36:38.087 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe41390b0 con 0x7febe413f0f0 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe4139390 con 0x7febe41461c0 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.082+0000 7febe9e46640 10 monclient(hunting): authenticate will time out at 3329.320038s 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febe413f0f0 0x7febe4136ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe37fe640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe40a8a20 0x7febe4144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe3fff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7febe41461c0 0x7febe41db080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febe413f0f0 0x7febe4136ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:42828/0 (socket says 172.21.15.67:42828) 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 1 -- 172.21.15.67:0/2487850331 learned_addr learned my addr 172.21.15.67:0/2487850331 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 10 monclient(hunting): get_auth_request con 0x7febe413f0f0 auth_method 0 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:38.088 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe37fe640 10 monclient(hunting): get_auth_request con 0x7febe40a8a20 auth_method 0 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe37fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe3fff640 10 monclient(hunting): get_auth_request con 0x7febe41461c0 auth_method 0 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe3fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.083+0000 7febe2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.084+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.084+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.084+0000 7febe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.084+0000 7febe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:38.089 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.084+0000 7febe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.084+0000 7febe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:38.090 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 10 monclient(hunting): handle_auth_done global_id 23248 payload 824 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 1 -- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7febe41461c0 msgr2=0x7febe41db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7febe41461c0 0x7febe41db080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 1 -- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe40a8a20 msgr2=0x7febe4144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe40a8a20 0x7febe4144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 10 monclient: _finish_hunting 0 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 1 monclient: found mon.noname-b 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.086+0000 7febe2ffd640 1 -- 172.21.15.67:0/2487850331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd8009400 con 0x7febe413f0f0 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.088+0000 7febe2ffd640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febe413f0f0 0x7febe4136ec0 secure :-1 s=READY pgs=5914 cs=0 l=1 rev1=1 crypto rx=0x7febd8000c00 tx=0x7febd80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.088+0000 7febcbfff640 10 client.?.objecter ms_handle_connect 0x7febe413f0f0 2023-12-01T17:36:38.091 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.088+0000 7febcbfff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 1 -- 172.21.15.67:0/2487850331 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd803a960 con 0x7febe413f0f0 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: dump: 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:38.093 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: _finish_auth 0 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 1 -- 172.21.15.67:0/2487850331 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd803aac0 con 0x7febe413f0f0 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 1 -- 172.21.15.67:0/2487850331 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd803ae30 con 0x7febe413f0f0 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 5 monclient: authenticate success, global_id 23248 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 10 monclient: _renew_subs 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe41da880 con 0x7febe413f0f0 2023-12-01T17:36:38.094 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febcbfff640 10 monclient: dump: 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe1ffb640 4 set_mon_vals no callback set 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 10 client.23248.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 10 monclient: _renew_subs 2023-12-01T17:36:38.095 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe41db5c0 con 0x7febe413f0f0 2023-12-01T17:36:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.092+0000 7febe9e46640 1 librados: init done 2023-12-01T17:36:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.093+0000 7febe9e46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.093+0000 7febe9e46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.093+0000 7febe9e46640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:38.096 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.093+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe414c820 con 0x7febe413f0f0 2023-12-01T17:36:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.111+0000 7febcbfff640 1 -- 172.21.15.67:0/2487850331 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7febd804d450 con 0x7febe413f0f0 2023-12-01T17:36:38.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.111+0000 7febcbfff640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7febd006c870 0x7febd006ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.111+0000 7febe37fe640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7febd006c870 0x7febd006ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.111+0000 7febe37fe640 10 monclient: get_auth_request con 0x7febd006c870 auth_method 0 2023-12-01T17:36:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.112+0000 7febcbfff640 10 client.23248.objecter ms_handle_connect 0x7febd006c870 2023-12-01T17:36:38.111 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.112+0000 7febe37fe640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7febd006c870 0x7febd006ed30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7febd4007fd0 tx=0x7febd4007e80 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.116+0000 7febcbfff640 1 -- 172.21.15.67:0/2487850331 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7febd807a7c0 con 0x7febe413f0f0 2023-12-01T17:36:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.116+0000 7febcbfff640 10 client.23248.objecter ms_dispatch 0x7febe4000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.116+0000 7febcbfff640 3 client.23248.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.116+0000 7febcbfff640 3 client.23248.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:38.116 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.116+0000 7febcbfff640 20 client.23248.objecter dump_active .. 0 homeless 2023-12-01T17:36:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.359+0000 7febcbfff640 1 -- 172.21.15.67:0/2487850331 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febd8076b00 con 0x7febe413f0f0 2023-12-01T17:36:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.359+0000 7febcbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:38.359 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.359+0000 7febcbfff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:38.537 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.537+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7febe413eb20 con 0x7febd006c870 2023-12-01T17:36:38.541 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.542+0000 7febcbfff640 1 -- 172.21.15.67:0/2487850331 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150836 (secure 0 0 0) 0x7febe413eb20 con 0x7febd006c870 2023-12-01T17:36:38.543 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:36:38.544 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 10 librados: watch_flush enter 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 10 librados: watch_flush exit 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 20 client.23248.objecter shutdown clearing up homeless session... 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 10 client.23248.objecter successfully canceled tick 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7febd006c870 msgr2=0x7febd006ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7febd006c870 0x7febd006ed30 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7febd4007fd0 tx=0x7febd4007e80 comp rx=0 tx=0).stop 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 10 monclient: shutdown 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febe413f0f0 msgr2=0x7febe4136ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febe413f0f0 0x7febe4136ec0 secure :-1 s=READY pgs=5914 cs=0 l=1 rev1=1 crypto rx=0x7febd8000c00 tx=0x7febd80046c0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 shutdown_connections 2023-12-01T17:36:38.548 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7febd006c870 0x7febd006ed30 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7febe41461c0 0x7febe41db080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7febe413f0f0 0x7febe4136ec0 unknown :-1 s=CLOSED pgs=5914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 --2- 172.21.15.67:0/2487850331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7febe40a8a20 0x7febe4144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 >> 172.21.15.67:0/2487850331 conn(0x7febe40b7890 msgr2=0x7febe413bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 shutdown_connections 2023-12-01T17:36:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 -- 172.21.15.67:0/2487850331 wait complete. 2023-12-01T17:36:38.549 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:38.544+0000 7febe9e46640 1 librados: shutdown 2023-12-01T17:36:38.554 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":2085,"stamp":"2023-12-01T17:36:36.775557+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":568788,"kb_used_data":53980,"kb_used_omap":12,"kb_used_meta":513139,"kb_avail":754405932,"statfs":{"total":773094113280,"available":772511674368,"internally_reserved":0,"allocated":55275520,"data_stored":30438252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":525454896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":56,"apply_latency_ms":56,"commit_latency_ns":56000000,"apply_latency_ns":56000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002014"},"pg_stats":[{"pgid":"22.6","version":"0'0","reported_seq":1049,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.093913+0000","last_change":"2023-12-01T17:05:14.287690+0000","last_active":"2023-12-01T17:36:26.093913+0000","last_peered":"2023-12-01T17:36:26.093913+0000","last_clean":"2023-12-01T17:36:26.093913+0000","last_became_active":"2023-12-01T17:05:14.257152+0000","last_became_peered":"2023-12-01T17:05:14.257152+0000","last_unstale":"2023-12-01T17:36:26.093913+0000","last_undegraded":"2023-12-01T17:36:26.093913+0000","last_fullsized":"2023-12-01T17:36:26.093913+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:59:48.591916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1057,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.100120+0000","last_change":"2023-12-01T17:05:21.303515+0000","last_active":"2023-12-01T17:36:26.100120+0000","last_peered":"2023-12-01T17:36:26.100120+0000","last_clean":"2023-12-01T17:36:26.100120+0000","last_became_active":"2023-12-01T17:05:21.296791+0000","last_became_peered":"2023-12-01T17:05:21.296791+0000","last_unstale":"2023-12-01T17:36:26.100120+0000","last_undegraded":"2023-12-01T17:36:26.100120+0000","last_fullsized":"2023-12-01T17:36:26.100120+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:51:36.558228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1056,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.138780+0000","last_change":"2023-12-01T17:05:13.270729+0000","last_active":"2023-12-01T17:36:26.138780+0000","last_peered":"2023-12-01T17:36:26.138780+0000","last_clean":"2023-12-01T17:36:26.138780+0000","last_became_active":"2023-12-01T17:05:13.265267+0000","last_became_peered":"2023-12-01T17:05:13.265267+0000","last_unstale":"2023-12-01T17:36:26.138780+0000","last_undegraded":"2023-12-01T17:36:26.138780+0000","last_fullsized":"2023-12-01T17:36:26.138780+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:57:02.423320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1054,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.260103+0000","last_change":"2023-12-01T17:05:38.668209+0000","last_active":"2023-12-01T17:36:27.260103+0000","last_peered":"2023-12-01T17:36:27.260103+0000","last_clean":"2023-12-01T17:36:27.260103+0000","last_became_active":"2023-12-01T17:05:38.651540+0000","last_became_peered":"2023-12-01T17:05:38.651540+0000","last_unstale":"2023-12-01T17:36:27.260103+0000","last_undegraded":"2023-12-01T17:36:27.260103+0000","last_fullsized":"2023-12-01T17:36:27.260103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1062,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.193251+0000","last_change":"2023-12-01T17:05:45.108671+0000","last_active":"2023-12-01T17:36:26.193251+0000","last_peered":"2023-12-01T17:36:26.193251+0000","last_clean":"2023-12-01T17:36:26.193251+0000","last_became_active":"2023-12-01T17:05:45.102496+0000","last_became_peered":"2023-12-01T17:05:45.102496+0000","last_unstale":"2023-12-01T17:36:26.193251+0000","last_undegraded":"2023-12-01T17:36:26.193251+0000","last_fullsized":"2023-12-01T17:36:26.193251+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:35:01.639680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1053,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.211957+0000","last_change":"2023-12-01T17:05:21.134670+0000","last_active":"2023-12-01T17:36:27.211957+0000","last_peered":"2023-12-01T17:36:27.211957+0000","last_clean":"2023-12-01T17:36:27.211957+0000","last_became_active":"2023-12-01T17:05:21.112377+0000","last_became_peered":"2023-12-01T17:05:21.112377+0000","last_unstale":"2023-12-01T17:36:27.211957+0000","last_undegraded":"2023-12-01T17:36:27.211957+0000","last_fullsized":"2023-12-01T17:36:27.211957+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.131517+0000","last_change":"2023-12-01T17:05:12.338713+0000","last_active":"2023-12-01T17:36:26.131517+0000","last_peered":"2023-12-01T17:36:26.131517+0000","last_clean":"2023-12-01T17:36:26.131517+0000","last_became_active":"2023-12-01T17:05:12.325443+0000","last_became_peered":"2023-12-01T17:05:12.325443+0000","last_unstale":"2023-12-01T17:36:26.131517+0000","last_undegraded":"2023-12-01T17:36:26.131517+0000","last_fullsized":"2023-12-01T17:36:26.131517+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:16:52.693300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1061,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:33.939249+0000","last_change":"2023-12-01T17:36:33.940179+0000","last_active":"2023-12-01T17:36:33.939249+0000","last_peered":"2023-12-01T17:36:33.939249+0000","last_clean":"2023-12-01T17:36:33.939249+0000","last_became_active":"2023-12-01T17:05:35.971286+0000","last_became_peered":"2023-12-01T17:05:35.971286+0000","last_unstale":"2023-12-01T17:36:33.939249+0000","last_undegraded":"2023-12-01T17:36:33.939249+0000","last_fullsized":"2023-12-01T17:36:33.939249+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:33.933861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:33.933861+0000","last_clean_scrub_stamp":"2023-12-01T17:36:33.933861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:29:13.183460+0000","scrub_duration":0.050039922000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1057,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:32.519091+0000","last_change":"2023-12-01T17:36:32.519854+0000","last_active":"2023-12-01T17:36:32.519091+0000","last_peered":"2023-12-01T17:36:32.519091+0000","last_clean":"2023-12-01T17:36:32.519091+0000","last_became_active":"2023-12-01T17:05:33.448448+0000","last_became_peered":"2023-12-01T17:05:33.448448+0000","last_unstale":"2023-12-01T17:36:32.519091+0000","last_undegraded":"2023-12-01T17:36:32.519091+0000","last_fullsized":"2023-12-01T17:36:32.519091+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:32.517436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:32.517436+0000","last_clean_scrub_stamp":"2023-12-01T17:36:32.517436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:57:35.099944+0000","scrub_duration":0.069663173999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1056,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.271820+0000","last_change":"2023-12-01T17:05:36.335708+0000","last_active":"2023-12-01T17:36:27.271820+0000","last_peered":"2023-12-01T17:36:27.271820+0000","last_clean":"2023-12-01T17:36:27.271820+0000","last_became_active":"2023-12-01T17:05:36.329117+0000","last_became_peered":"2023-12-01T17:05:36.329117+0000","last_unstale":"2023-12-01T17:36:27.271820+0000","last_undegraded":"2023-12-01T17:36:27.271820+0000","last_fullsized":"2023-12-01T17:36:27.271820+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.140840+0000","last_change":"2023-12-01T17:05:11.457400+0000","last_active":"2023-12-01T17:36:26.140840+0000","last_peered":"2023-12-01T17:36:26.140840+0000","last_clean":"2023-12-01T17:36:26.140840+0000","last_became_active":"2023-12-01T17:05:11.452081+0000","last_became_peered":"2023-12-01T17:05:11.452081+0000","last_unstale":"2023-12-01T17:36:26.140840+0000","last_undegraded":"2023-12-01T17:36:26.140840+0000","last_fullsized":"2023-12-01T17:36:26.140840+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:10:31.762918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1055,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.065933+0000","last_change":"2023-12-01T17:05:36.195013+0000","last_active":"2023-12-01T17:36:26.065933+0000","last_peered":"2023-12-01T17:36:26.065933+0000","last_clean":"2023-12-01T17:36:26.065933+0000","last_became_active":"2023-12-01T17:05:36.185378+0000","last_became_peered":"2023-12-01T17:05:36.185378+0000","last_unstale":"2023-12-01T17:36:26.065933+0000","last_undegraded":"2023-12-01T17:36:26.065933+0000","last_fullsized":"2023-12-01T17:36:26.065933+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:38:11.192241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1099'78","reported_seq":1174,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:34.048890+0000","last_change":"2023-12-01T17:36:34.050160+0000","last_active":"2023-12-01T17:36:34.048890+0000","last_peered":"2023-12-01T17:36:34.048890+0000","last_clean":"2023-12-01T17:36:34.048890+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:36:34.048890+0000","last_undegraded":"2023-12-01T17:36:34.048890+0000","last_fullsized":"2023-12-01T17:36:34.048890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'78","last_scrub_stamp":"2023-12-01T17:36:34.044757+0000","last_deep_scrub":"1099'78","last_deep_scrub_stamp":"2023-12-01T17:36:34.044757+0000","last_clean_scrub_stamp":"2023-12-01T17:36:34.044757+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:16:14.393778+0000","scrub_duration":0.143069317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1053,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.061599+0000","last_change":"2023-12-01T17:04:01.495183+0000","last_active":"2023-12-01T17:36:26.061599+0000","last_peered":"2023-12-01T17:36:26.061599+0000","last_clean":"2023-12-01T17:36:26.061599+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:36:26.061599+0000","last_undegraded":"2023-12-01T17:36:26.061599+0000","last_fullsized":"2023-12-01T17:36:26.061599+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:13:19.704857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1049,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.318088+0000","last_change":"2023-12-01T17:05:15.674793+0000","last_active":"2023-12-01T17:36:27.318088+0000","last_peered":"2023-12-01T17:36:27.318088+0000","last_clean":"2023-12-01T17:36:27.318088+0000","last_became_active":"2023-12-01T17:05:15.659631+0000","last_became_peered":"2023-12-01T17:05:15.659631+0000","last_unstale":"2023-12-01T17:36:27.318088+0000","last_undegraded":"2023-12-01T17:36:27.318088+0000","last_fullsized":"2023-12-01T17:36:27.318088+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1055,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.035615+0000","last_change":"2023-12-01T17:04:01.146575+0000","last_active":"2023-12-01T17:36:26.035615+0000","last_peered":"2023-12-01T17:36:26.035615+0000","last_clean":"2023-12-01T17:36:26.035615+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:36:26.035615+0000","last_undegraded":"2023-12-01T17:36:26.035615+0000","last_fullsized":"2023-12-01T17:36:26.035615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:18:50.459254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1052,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.282395+0000","last_change":"2023-12-01T17:05:11.335421+0000","last_active":"2023-12-01T17:36:27.282395+0000","last_peered":"2023-12-01T17:36:27.282395+0000","last_clean":"2023-12-01T17:36:27.282395+0000","last_became_active":"2023-12-01T17:05:11.329382+0000","last_became_peered":"2023-12-01T17:05:11.329382+0000","last_unstale":"2023-12-01T17:36:27.282395+0000","last_undegraded":"2023-12-01T17:36:27.282395+0000","last_fullsized":"2023-12-01T17:36:27.282395+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1061,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.025305+0000","last_change":"2023-12-01T17:04:01.107520+0000","last_active":"2023-12-01T17:36:26.025305+0000","last_peered":"2023-12-01T17:36:26.025305+0000","last_clean":"2023-12-01T17:36:26.025305+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:36:26.025305+0000","last_undegraded":"2023-12-01T17:36:26.025305+0000","last_fullsized":"2023-12-01T17:36:26.025305+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:47:47.456810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1054,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.477235+0000","last_change":"2023-12-01T17:05:16.493438+0000","last_active":"2023-12-01T17:36:27.477235+0000","last_peered":"2023-12-01T17:36:27.477235+0000","last_clean":"2023-12-01T17:36:27.477235+0000","last_became_active":"2023-12-01T17:05:16.481672+0000","last_became_peered":"2023-12-01T17:05:16.481672+0000","last_unstale":"2023-12-01T17:36:27.477235+0000","last_undegraded":"2023-12-01T17:36:27.477235+0000","last_fullsized":"2023-12-01T17:36:27.477235+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:22:21.064006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.050632+0000","last_change":"2023-12-01T17:04:01.438358+0000","last_active":"2023-12-01T17:36:26.050632+0000","last_peered":"2023-12-01T17:36:26.050632+0000","last_clean":"2023-12-01T17:36:26.050632+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:36:26.050632+0000","last_undegraded":"2023-12-01T17:36:26.050632+0000","last_fullsized":"2023-12-01T17:36:26.050632+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:45:24.963538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1055,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.507403+0000","last_change":"2023-12-01T17:05:14.953876+0000","last_active":"2023-12-01T17:36:27.507403+0000","last_peered":"2023-12-01T17:36:27.507403+0000","last_clean":"2023-12-01T17:36:27.507403+0000","last_became_active":"2023-12-01T17:05:14.948506+0000","last_became_peered":"2023-12-01T17:05:14.948506+0000","last_unstale":"2023-12-01T17:36:27.507403+0000","last_undegraded":"2023-12-01T17:36:27.507403+0000","last_fullsized":"2023-12-01T17:36:27.507403+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:52.139356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1060,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.137101+0000","last_change":"2023-12-01T17:04:01.467571+0000","last_active":"2023-12-01T17:36:26.137101+0000","last_peered":"2023-12-01T17:36:26.137101+0000","last_clean":"2023-12-01T17:36:26.137101+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:36:26.137101+0000","last_undegraded":"2023-12-01T17:36:26.137101+0000","last_fullsized":"2023-12-01T17:36:26.137101+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:41:15.693504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1052,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.180117+0000","last_change":"2023-12-01T17:05:11.984847+0000","last_active":"2023-12-01T17:36:27.180117+0000","last_peered":"2023-12-01T17:36:27.180117+0000","last_clean":"2023-12-01T17:36:27.180117+0000","last_became_active":"2023-12-01T17:05:11.693016+0000","last_became_peered":"2023-12-01T17:05:11.693016+0000","last_unstale":"2023-12-01T17:36:27.180117+0000","last_undegraded":"2023-12-01T17:36:27.180117+0000","last_fullsized":"2023-12-01T17:36:27.180117+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1054,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.095975+0000","last_change":"2023-12-01T17:04:01.201853+0000","last_active":"2023-12-01T17:36:26.095975+0000","last_peered":"2023-12-01T17:36:26.095975+0000","last_clean":"2023-12-01T17:36:26.095975+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:36:26.095975+0000","last_undegraded":"2023-12-01T17:36:26.095975+0000","last_fullsized":"2023-12-01T17:36:26.095975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:38:19.219620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1066,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.032799+0000","last_change":"2023-12-01T17:05:12.792097+0000","last_active":"2023-12-01T17:36:26.032799+0000","last_peered":"2023-12-01T17:36:26.032799+0000","last_clean":"2023-12-01T17:36:26.032799+0000","last_became_active":"2023-12-01T17:05:12.768446+0000","last_became_peered":"2023-12-01T17:05:12.768446+0000","last_unstale":"2023-12-01T17:36:26.032799+0000","last_undegraded":"2023-12-01T17:36:26.032799+0000","last_fullsized":"2023-12-01T17:36:26.032799+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:17:04.335014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1068,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.524284+0000","last_change":"2023-12-01T17:04:01.536883+0000","last_active":"2023-12-01T17:36:27.524284+0000","last_peered":"2023-12-01T17:36:27.524284+0000","last_clean":"2023-12-01T17:36:27.524284+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:36:27.524284+0000","last_undegraded":"2023-12-01T17:36:27.524284+0000","last_fullsized":"2023-12-01T17:36:27.524284+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:42:47.593805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1057,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.032549+0000","last_change":"2023-12-01T17:05:24.990862+0000","last_active":"2023-12-01T17:36:26.032549+0000","last_peered":"2023-12-01T17:36:26.032549+0000","last_clean":"2023-12-01T17:36:26.032549+0000","last_became_active":"2023-12-01T17:05:24.924547+0000","last_became_peered":"2023-12-01T17:05:24.924547+0000","last_unstale":"2023-12-01T17:36:26.032549+0000","last_undegraded":"2023-12-01T17:36:26.032549+0000","last_fullsized":"2023-12-01T17:36:26.032549+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:04:58.899609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1050,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.113311+0000","last_change":"2023-12-01T17:04:01.476480+0000","last_active":"2023-12-01T17:36:26.113311+0000","last_peered":"2023-12-01T17:36:26.113311+0000","last_clean":"2023-12-01T17:36:26.113311+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:36:26.113311+0000","last_undegraded":"2023-12-01T17:36:26.113311+0000","last_fullsized":"2023-12-01T17:36:26.113311+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:03:59.560899+0000","last_clean_scrub_stamp":"2023-12-01T17:03:59.560899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:26:16.264231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1063,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.195373+0000","last_change":"2023-12-01T17:05:28.757137+0000","last_active":"2023-12-01T17:36:26.195373+0000","last_peered":"2023-12-01T17:36:26.195373+0000","last_clean":"2023-12-01T17:36:26.195373+0000","last_became_active":"2023-12-01T17:05:28.748541+0000","last_became_peered":"2023-12-01T17:05:28.748541+0000","last_unstale":"2023-12-01T17:36:26.195373+0000","last_undegraded":"2023-12-01T17:36:26.195373+0000","last_fullsized":"2023-12-01T17:36:26.195373+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:15:33.293632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1","version":"62'7","reported_seq":1069,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.201851+0000","last_change":"2023-12-01T17:05:38.783874+0000","last_active":"2023-12-01T17:36:27.201851+0000","last_peered":"2023-12-01T17:36:27.201851+0000","last_clean":"2023-12-01T17:36:27.201851+0000","last_became_active":"2023-12-01T17:05:38.775069+0000","last_became_peered":"2023-12-01T17:05:38.775069+0000","last_unstale":"2023-12-01T17:36:27.201851+0000","last_undegraded":"2023-12-01T17:36:27.201851+0000","last_fullsized":"2023-12-01T17:36:27.201851+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1057,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.194366+0000","last_change":"2023-12-01T17:05:17.531869+0000","last_active":"2023-12-01T17:36:27.194366+0000","last_peered":"2023-12-01T17:36:27.194366+0000","last_clean":"2023-12-01T17:36:27.194366+0000","last_became_active":"2023-12-01T17:05:17.447969+0000","last_became_peered":"2023-12-01T17:05:17.447969+0000","last_unstale":"2023-12-01T17:36:27.194366+0000","last_undegraded":"2023-12-01T17:36:27.194366+0000","last_fullsized":"2023-12-01T17:36:27.194366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1054,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:29.208250+0000","last_change":"2023-12-01T17:05:12.380756+0000","last_active":"2023-12-01T17:36:29.208250+0000","last_peered":"2023-12-01T17:36:29.208250+0000","last_clean":"2023-12-01T17:36:29.208250+0000","last_became_active":"2023-12-01T17:05:12.334114+0000","last_became_peered":"2023-12-01T17:05:12.334114+0000","last_unstale":"2023-12-01T17:36:29.208250+0000","last_undegraded":"2023-12-01T17:36:29.208250+0000","last_fullsized":"2023-12-01T17:36:29.208250+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1068,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.038993+0000","last_change":"2023-12-01T17:05:39.346000+0000","last_active":"2023-12-01T17:36:26.038993+0000","last_peered":"2023-12-01T17:36:26.038993+0000","last_clean":"2023-12-01T17:36:26.038993+0000","last_became_active":"2023-12-01T17:05:39.340484+0000","last_became_peered":"2023-12-01T17:05:39.340484+0000","last_unstale":"2023-12-01T17:36:26.038993+0000","last_undegraded":"2023-12-01T17:36:26.038993+0000","last_fullsized":"2023-12-01T17:36:26.038993+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:50:46.835583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1051,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.311569+0000","last_change":"2023-12-01T17:05:58.777840+0000","last_active":"2023-12-01T17:36:27.311569+0000","last_peered":"2023-12-01T17:36:27.311569+0000","last_clean":"2023-12-01T17:36:27.311569+0000","last_became_active":"2023-12-01T17:05:58.772230+0000","last_became_peered":"2023-12-01T17:05:58.772230+0000","last_unstale":"2023-12-01T17:36:27.311569+0000","last_undegraded":"2023-12-01T17:36:27.311569+0000","last_fullsized":"2023-12-01T17:36:27.311569+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1048,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.080987+0000","last_change":"2023-12-01T17:05:11.225296+0000","last_active":"2023-12-01T17:36:26.080987+0000","last_peered":"2023-12-01T17:36:26.080987+0000","last_clean":"2023-12-01T17:36:26.080987+0000","last_became_active":"2023-12-01T17:05:11.209283+0000","last_became_peered":"2023-12-01T17:05:11.209283+0000","last_unstale":"2023-12-01T17:36:26.080987+0000","last_undegraded":"2023-12-01T17:36:26.080987+0000","last_fullsized":"2023-12-01T17:36:26.080987+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:39:18.388303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1051,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.320112+0000","last_change":"2023-12-01T17:05:17.271434+0000","last_active":"2023-12-01T17:36:27.320112+0000","last_peered":"2023-12-01T17:36:27.320112+0000","last_clean":"2023-12-01T17:36:27.320112+0000","last_became_active":"2023-12-01T17:05:17.215461+0000","last_became_peered":"2023-12-01T17:05:17.215461+0000","last_unstale":"2023-12-01T17:36:27.320112+0000","last_undegraded":"2023-12-01T17:36:27.320112+0000","last_fullsized":"2023-12-01T17:36:27.320112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1047,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.526037+0000","last_change":"2023-12-01T17:05:21.697504+0000","last_active":"2023-12-01T17:36:27.526037+0000","last_peered":"2023-12-01T17:36:27.526037+0000","last_clean":"2023-12-01T17:36:27.526037+0000","last_became_active":"2023-12-01T17:05:21.668214+0000","last_became_peered":"2023-12-01T17:05:21.668214+0000","last_unstale":"2023-12-01T17:36:27.526037+0000","last_undegraded":"2023-12-01T17:36:27.526037+0000","last_fullsized":"2023-12-01T17:36:27.526037+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1064,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.170472+0000","last_change":"2023-12-01T17:05:46.022392+0000","last_active":"2023-12-01T17:36:26.170472+0000","last_peered":"2023-12-01T17:36:26.170472+0000","last_clean":"2023-12-01T17:36:26.170472+0000","last_became_active":"2023-12-01T17:05:46.016610+0000","last_became_peered":"2023-12-01T17:05:46.016610+0000","last_unstale":"2023-12-01T17:36:26.170472+0000","last_undegraded":"2023-12-01T17:36:26.170472+0000","last_fullsized":"2023-12-01T17:36:26.170472+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:47:36.167822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1050,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.032757+0000","last_change":"2023-12-01T17:05:37.365267+0000","last_active":"2023-12-01T17:36:26.032757+0000","last_peered":"2023-12-01T17:36:26.032757+0000","last_clean":"2023-12-01T17:36:26.032757+0000","last_became_active":"2023-12-01T17:05:37.358444+0000","last_became_peered":"2023-12-01T17:05:37.358444+0000","last_unstale":"2023-12-01T17:36:26.032757+0000","last_undegraded":"2023-12-01T17:36:26.032757+0000","last_fullsized":"2023-12-01T17:36:26.032757+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:48:13.502545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1055,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.162048+0000","last_change":"2023-12-01T17:05:13.509587+0000","last_active":"2023-12-01T17:36:26.162048+0000","last_peered":"2023-12-01T17:36:26.162048+0000","last_clean":"2023-12-01T17:36:26.162048+0000","last_became_active":"2023-12-01T17:05:13.499129+0000","last_became_peered":"2023-12-01T17:05:13.499129+0000","last_unstale":"2023-12-01T17:36:26.162048+0000","last_undegraded":"2023-12-01T17:36:26.162048+0000","last_fullsized":"2023-12-01T17:36:26.162048+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:25:34.541978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1050,"reported_epoch":1098,"state":"active+clean","last_fresh":"2023-12-01T17:36:26.052864+0000","last_change":"2023-12-01T17:05:37.748101+0000","last_active":"2023-12-01T17:36:26.052864+0000","last_peered":"2023-12-01T17:36:26.052864+0000","last_clean":"2023-12-01T17:36:26.052864+0000","last_became_active":"2023-12-01T17:05:37.740514+0000","last_became_peered":"2023-12-01T17:05:37.740514+0000","last_unstale":"2023-12-01T17:36:26.052864+0000","last_undegraded":"2023-12-01T17:36:26.052864+0000","last_fullsized":"2023-12-01T17:36:26.052864+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T04:17:18.757175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444423,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70368,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301472,"statfs":{"total":96636764160,"available":96564707328,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":20.553999999999998,"5min":12.125999999999999,"15min":15.801},"min":{"1min":6.5620000000000003,"5min":4.5899999999999999,"15min":4.5899999999999999},"max":{"1min":127.741,"5min":127.741,"15min":150.279},"last":16.687000000000001},{"interface":"front","average":{"1min":29.547000000000001,"5min":14.099,"15min":17.922999999999998},"min":{"1min":8.5419999999999998,"5min":4.7229999999999999,"15min":4.7229999999999999},"max":{"1min":184.30600000000001,"5min":184.30600000000001,"15min":184.30600000000001},"last":12.890000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":18.321999999999999,"5min":12.057,"15min":16.654},"min":{"1min":6.5709999999999997,"5min":5.4610000000000003,"15min":5.4610000000000003},"max":{"1min":99.156000000000006,"5min":99.156000000000006,"15min":208.18199999999999},"last":8.6620000000000008},{"interface":"front","average":{"1min":34.377000000000002,"5min":16.986999999999998,"15min":19.702000000000002},"min":{"1min":7.9260000000000002,"5min":6.8769999999999998,"15min":6.2030000000000003},"max":{"1min":216.785,"5min":216.785,"15min":216.785},"last":19.154}]},{"osd":3,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":28.09,"5min":16.707999999999998,"15min":21.498000000000001},"min":{"1min":8.9960000000000004,"5min":7.8550000000000004,"15min":7.8550000000000004},"max":{"1min":243.88999999999999,"5min":243.88999999999999,"15min":276.07400000000001},"last":18.164999999999999},{"interface":"front","average":{"1min":30.187000000000001,"5min":19.353999999999999,"15min":24.324000000000002},"min":{"1min":13.336,"5min":10.973000000000001,"15min":7.2649999999999997},"max":{"1min":239.40000000000001,"5min":239.40000000000001,"15min":252.43299999999999},"last":21.445}]},{"osd":4,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":31.789000000000001,"5min":16.448,"15min":17.77},"min":{"1min":8.0429999999999993,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":230.511,"5min":230.511,"15min":230.511},"last":22.581},{"interface":"front","average":{"1min":33.975000000000001,"5min":17.349,"15min":21.777999999999999},"min":{"1min":7.6379999999999999,"5min":7.3979999999999997,"15min":7.3979999999999997},"max":{"1min":241.28100000000001,"5min":241.28100000000001,"15min":241.28100000000001},"last":12.259}]},{"osd":5,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":27.834,"5min":16.713999999999999,"15min":18.786000000000001},"min":{"1min":6.859,"5min":6.859,"15min":6.149},"max":{"1min":191.78700000000001,"5min":191.78700000000001,"15min":191.78700000000001},"last":28.832999999999998},{"interface":"front","average":{"1min":17.850999999999999,"5min":15.477,"15min":22.440999999999999},"min":{"1min":7.3769999999999998,"5min":7.3769999999999998,"15min":7.0270000000000001},"max":{"1min":111.208,"5min":111.208,"15min":311.13299999999998},"last":26.872}]},{"osd":6,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":22.614999999999998,"5min":17.094999999999999,"15min":18.754999999999999},"min":{"1min":11.262,"5min":10.199,"15min":7.0960000000000001},"max":{"1min":94.787999999999997,"5min":94.787999999999997,"15min":99.623000000000005},"last":20.23},{"interface":"front","average":{"1min":26.064,"5min":17.611000000000001,"15min":21.722999999999999},"min":{"1min":8.6600000000000001,"5min":8.6080000000000005,"15min":7.9909999999999997},"max":{"1min":153.83799999999999,"5min":153.83799999999999,"15min":186.11000000000001},"last":11.726000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":35.716000000000001,"5min":20.347000000000001,"15min":24.768999999999998},"min":{"1min":12.414999999999999,"5min":9.4819999999999993,"15min":9.1150000000000002},"max":{"1min":207.64599999999999,"5min":207.64599999999999,"15min":207.64599999999999},"last":23.23},{"interface":"front","average":{"1min":23.535,"5min":17.969999999999999,"15min":21.334},"min":{"1min":11.978999999999999,"5min":9.4469999999999992,"15min":9.0069999999999997},"max":{"1min":74.777000000000001,"5min":74.777000000000001,"15min":148.89599999999999},"last":14.964}]}]},{"osd":3,"up_from":16,"seq":68719477127,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69600,"kb_used_data":6488,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302240,"statfs":{"total":96636764160,"available":96565493760,"internally_reserved":0,"allocated":6643712,"data_stored":3599218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":38.427,"5min":15.26,"15min":18.015000000000001},"min":{"1min":6.8440000000000003,"5min":5.5540000000000003,"15min":5.2539999999999996},"max":{"1min":180.16,"5min":180.16,"15min":181.94200000000001},"last":6.6379999999999999},{"interface":"front","average":{"1min":28.507000000000001,"5min":13.943,"15min":18.838000000000001},"min":{"1min":4.915,"5min":4.915,"15min":4.915},"max":{"1min":107.5,"5min":107.5,"15min":427.36500000000001},"last":9.577}]},{"osd":1,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":30.577999999999999,"5min":15.76,"15min":20.832999999999998},"min":{"1min":8.1419999999999995,"5min":7.585,"15min":6.4039999999999999},"max":{"1min":146.48099999999999,"5min":146.48099999999999,"15min":227.309},"last":13.561999999999999},{"interface":"front","average":{"1min":33.158000000000001,"5min":15.33,"15min":19.213999999999999},"min":{"1min":8.2870000000000008,"5min":6.9850000000000003,"15min":5.827},"max":{"1min":223.179,"5min":223.179,"15min":421.40600000000001},"last":7.5590000000000002}]},{"osd":2,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":20.815999999999999,"5min":13.278,"15min":22.062000000000001},"min":{"1min":7.9829999999999997,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":86.969999999999999,"5min":86.969999999999999,"15min":451.97899999999998},"last":12.811999999999999},{"interface":"front","average":{"1min":29.763999999999999,"5min":15.894,"15min":23.613},"min":{"1min":6.8540000000000001,"5min":6.7190000000000003,"15min":6.5170000000000003},"max":{"1min":223.84,"5min":223.84,"15min":384.58699999999999},"last":10.833}]},{"osd":4,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":33.475999999999999,"5min":17.899999999999999,"15min":21.863},"min":{"1min":9.6440000000000001,"5min":8.5640000000000001,"15min":6.5330000000000004},"max":{"1min":244.989,"5min":244.989,"15min":424.09500000000003},"last":11.436999999999999},{"interface":"front","average":{"1min":31.292000000000002,"5min":17.065000000000001,"15min":21.056000000000001},"min":{"1min":8.9700000000000006,"5min":7.9870000000000001,"15min":7.0049999999999999},"max":{"1min":159.78200000000001,"5min":159.78200000000001,"15min":202.70500000000001},"last":17.195}]},{"osd":5,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":32.497,"5min":19.154,"15min":25.114000000000001},"min":{"1min":10.481999999999999,"5min":9.7400000000000002,"15min":9.0860000000000003},"max":{"1min":228.96700000000001,"5min":228.96700000000001,"15min":424.80099999999999},"last":12.031000000000001},{"interface":"front","average":{"1min":23.297999999999998,"5min":15.457000000000001,"15min":21.076000000000001},"min":{"1min":8.0359999999999996,"5min":7.2850000000000001,"15min":7.1710000000000003},"max":{"1min":87.635000000000005,"5min":87.635000000000005,"15min":231.65199999999999},"last":14.282999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":27.949000000000002,"5min":17.577000000000002,"15min":21.687999999999999},"min":{"1min":8.2859999999999996,"5min":8.2859999999999996,"15min":7.9340000000000002},"max":{"1min":136.684,"5min":136.684,"15min":203.99100000000001},"last":18.879000000000001},{"interface":"front","average":{"1min":30.741,"5min":19.934000000000001,"15min":25.875},"min":{"1min":12.468999999999999,"5min":10.978,"15min":10.978},"max":{"1min":147.88200000000001,"5min":147.88200000000001,"15min":203.34100000000001},"last":18.068999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":25.036000000000001,"5min":16.992999999999999,"15min":22.780000000000001},"min":{"1min":8.7509999999999994,"5min":7.9589999999999996,"15min":7.0259999999999998},"max":{"1min":105.342,"5min":105.342,"15min":196.05699999999999},"last":10.228},{"interface":"front","average":{"1min":27.623000000000001,"5min":19.024999999999999,"15min":25.555},"min":{"1min":11.186,"5min":11.186,"15min":10.492000000000001},"max":{"1min":102.29000000000001,"5min":102.29000000000001,"15min":236.84700000000001},"last":14.920999999999999}]}]},{"osd":2,"up_from":16,"seq":68719477125,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70412,"kb_used_data":6492,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301428,"statfs":{"total":96636764160,"available":96564662272,"internally_reserved":0,"allocated":6647808,"data_stored":3599223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.219000000000001,"5min":14.260999999999999,"15min":20.582999999999998},"min":{"1min":6.4989999999999997,"5min":4.7110000000000003,"15min":4.6059999999999999},"max":{"1min":169.971,"5min":169.971,"15min":231.76499999999999},"last":17.425000000000001},{"interface":"front","average":{"1min":29.649999999999999,"5min":14.412000000000001,"15min":19.628},"min":{"1min":5.3049999999999997,"5min":5.1890000000000001,"15min":4.8810000000000002},"max":{"1min":150.74199999999999,"5min":150.74199999999999,"15min":222.62},"last":14.693}]},{"osd":1,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":23.131,"5min":12.741,"15min":20.030000000000001},"min":{"1min":6.4980000000000002,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":182.83600000000001,"5min":182.83600000000001,"15min":220.96700000000001},"last":10.888999999999999},{"interface":"front","average":{"1min":27.039999999999999,"5min":16.943999999999999,"15min":23.343},"min":{"1min":8.6869999999999994,"5min":8.6869999999999994,"15min":7.8639999999999999},"max":{"1min":196.886,"5min":196.886,"15min":213.27799999999999},"last":21.724}]},{"osd":3,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":40.603999999999999,"5min":17.788,"15min":22.448},"min":{"1min":7.0609999999999999,"5min":7.0609999999999999,"15min":6.734},"max":{"1min":177.04900000000001,"5min":177.04900000000001,"15min":219.51300000000001},"last":9.6750000000000007},{"interface":"front","average":{"1min":41.942999999999998,"5min":19.474,"15min":24.347000000000001},"min":{"1min":7.6740000000000004,"5min":7.6740000000000004,"15min":6.9900000000000002},"max":{"1min":204.721,"5min":204.721,"15min":204.721},"last":11.632999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.542000000000002,"5min":16.068000000000001,"15min":21.274000000000001},"min":{"1min":8.6310000000000002,"5min":7.2060000000000004,"15min":6.6799999999999997},"max":{"1min":180.369,"5min":180.369,"15min":251.745},"last":13.786},{"interface":"front","average":{"1min":33.243000000000002,"5min":18.972000000000001,"15min":24.882999999999999},"min":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":7.5720000000000001},"max":{"1min":170.637,"5min":170.637,"15min":287.32100000000003},"last":19.370999999999999}]},{"osd":5,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":37.953000000000003,"5min":18.436,"15min":24.466999999999999},"min":{"1min":8.5060000000000002,"5min":8.3300000000000001,"15min":8.0020000000000007},"max":{"1min":206.56700000000001,"5min":206.56700000000001,"15min":210.012},"last":16.173999999999999},{"interface":"front","average":{"1min":33.348999999999997,"5min":19.402000000000001,"15min":25.024000000000001},"min":{"1min":9.1980000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"max":{"1min":232.91900000000001,"5min":232.91900000000001,"15min":232.91900000000001},"last":21.088999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":35.573999999999998,"5min":19.463000000000001,"15min":23.882999999999999},"min":{"1min":8.8710000000000004,"5min":8.8710000000000004,"15min":8.1829999999999998},"max":{"1min":182.07400000000001,"5min":182.07400000000001,"15min":237.21899999999999},"last":16.829000000000001},{"interface":"front","average":{"1min":38.033999999999999,"5min":20.099,"15min":27.759},"min":{"1min":12.598000000000001,"5min":9.5960000000000001,"15min":9.5649999999999995},"max":{"1min":185.90299999999999,"5min":185.90299999999999,"15min":241.92699999999999},"last":20.335999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":19.201000000000001,"5min":16.905000000000001,"15min":24.222000000000001},"min":{"1min":8.7940000000000005,"5min":8.7940000000000005,"15min":8.5690000000000008},"max":{"1min":68.075000000000003,"5min":68.075000000000003,"15min":341.73000000000002},"last":8.9269999999999996},{"interface":"front","average":{"1min":22.43,"5min":18.937999999999999,"15min":27.407},"min":{"1min":10.853,"5min":10.853,"15min":10.853},"max":{"1min":89.631,"5min":89.631,"15min":343.17399999999998},"last":18.341999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69972,"kb_used_data":6860,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301868,"statfs":{"total":96636764160,"available":96565112832,"internally_reserved":0,"allocated":7024640,"data_stored":3846006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 17:36:15 2023","interfaces":[{"interface":"back","average":{"1min":11.167,"5min":10.766999999999999,"15min":17.079000000000001},"min":{"1min":5.5579999999999998,"5min":5.5579999999999998,"15min":4.3550000000000004},"max":{"1min":33.322000000000003,"5min":33.322000000000003,"15min":210.43299999999999},"last":12.497999999999999},{"interface":"front","average":{"1min":12.661,"5min":10.788,"15min":18.704999999999998},"min":{"1min":6.242,"5min":6.0750000000000002,"15min":5.2590000000000003},"max":{"1min":44.020000000000003,"5min":44.020000000000003,"15min":257.30500000000001},"last":8.2230000000000008}]},{"osd":2,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.808,"5min":14.16,"15min":19.648},"min":{"1min":8.2010000000000005,"5min":7.4740000000000002,"15min":6.452},"max":{"1min":94.155000000000001,"5min":94.155000000000001,"15min":239.24600000000001},"last":16.059000000000001},{"interface":"front","average":{"1min":16.765000000000001,"5min":13.634,"15min":18.114999999999998},"min":{"1min":8.3179999999999996,"5min":6.8579999999999997,"15min":5.6520000000000001},"max":{"1min":86.099999999999994,"5min":86.099999999999994,"15min":246.28299999999999},"last":14.189}]},{"osd":3,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":12.295999999999999,"15min":19.577000000000002},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"max":{"1min":128.08099999999999,"5min":128.08099999999999,"15min":240.017},"last":7.7990000000000004},{"interface":"front","average":{"1min":28.396000000000001,"5min":17.071999999999999,"15min":25.015999999999998},"min":{"1min":9.452,"5min":8.3499999999999996,"15min":7.2779999999999996},"max":{"1min":136.40600000000001,"5min":136.40600000000001,"15min":249.86099999999999},"last":11.750999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":15.007999999999999,"5min":12.295,"15min":18.792999999999999},"min":{"1min":7.5780000000000003,"5min":6.6909999999999998,"15min":6.6909999999999998},"max":{"1min":77.429000000000002,"5min":77.429000000000002,"15min":241.292},"last":8.8040000000000003},{"interface":"front","average":{"1min":19.893999999999998,"5min":16.827000000000002,"15min":22.696000000000002},"min":{"1min":11.875,"5min":9.3759999999999994,"15min":7.2290000000000001},"max":{"1min":91.792000000000002,"5min":91.792000000000002,"15min":236.65799999999999},"last":18.603000000000002}]},{"osd":5,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":16.934000000000001,"5min":14.622,"15min":24.321999999999999},"min":{"1min":9.2949999999999999,"5min":8.6929999999999996,"15min":7.9400000000000004},"max":{"1min":45.228000000000002,"5min":45.228000000000002,"15min":241.91200000000001},"last":17.041},{"interface":"front","average":{"1min":16.408000000000001,"5min":14.084,"15min":21.462},"min":{"1min":7.2779999999999996,"5min":7.2779999999999996,"15min":6.4279999999999999},"max":{"1min":66.477000000000004,"5min":66.477000000000004,"15min":247.185},"last":23.25}]},{"osd":6,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":15.752000000000001,"15min":23.873999999999999},"min":{"1min":9.8810000000000002,"5min":8.2829999999999995,"15min":7.923},"max":{"1min":50.194000000000003,"5min":50.194000000000003,"15min":244.93799999999999},"last":10.632},{"interface":"front","average":{"1min":16.581,"5min":15.106999999999999,"15min":22.677},"min":{"1min":9.2609999999999992,"5min":9.2110000000000003,"15min":7.734},"max":{"1min":42.387999999999998,"5min":42.387999999999998,"15min":247.80500000000001},"last":17.638999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":18.469999999999999,"5min":17.091999999999999,"15min":24.052},"min":{"1min":11.074999999999999,"5min":10.281000000000001,"15min":9.0820000000000007},"max":{"1min":37.758000000000003,"5min":37.758000000000003,"15min":238.58099999999999},"last":13.359},{"interface":"front","average":{"1min":18.225000000000001,"5min":16.898,"15min":24.481000000000002},"min":{"1min":11.706,"5min":9.3140000000000001,"15min":9.3140000000000001},"max":{"1min":36.850000000000001,"5min":36.850000000000001,"15min":273.142},"last":21.978999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542536,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69788,"kb_used_data":6492,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302052,"statfs":{"total":96636764160,"available":96565301248,"internally_reserved":0,"allocated":6647808,"data_stored":3599223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":10.34,"5min":9.7390000000000008,"15min":13.387},"min":{"1min":4.8339999999999996,"5min":4.577,"15min":4.0730000000000004},"max":{"1min":25.731999999999999,"5min":25.731999999999999,"15min":129.89699999999999},"last":5.1239999999999997},{"interface":"front","average":{"1min":11.954000000000001,"5min":10.832000000000001,"15min":15.518000000000001},"min":{"1min":6.5010000000000003,"5min":6.4669999999999996,"15min":4.8129999999999997},"max":{"1min":34.526000000000003,"5min":34.526000000000003,"15min":155.99100000000001},"last":12.657}]},{"osd":1,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":14.457000000000001,"5min":12.138,"15min":15.375},"min":{"1min":7.9969999999999999,"5min":6.702,"15min":6.2789999999999999},"max":{"1min":50.130000000000003,"5min":50.130000000000003,"15min":175.55000000000001},"last":6.9640000000000004},{"interface":"front","average":{"1min":18.449000000000002,"5min":15.638999999999999,"15min":19.158000000000001},"min":{"1min":10.378,"5min":10.378,"15min":7.0490000000000004},"max":{"1min":54.060000000000002,"5min":54.060000000000002,"15min":130.91999999999999},"last":10.702999999999999}]},{"osd":2,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":21.776,"5min":15.327,"15min":19.893999999999998},"min":{"1min":7.0010000000000003,"5min":6.8879999999999999,"15min":6.8879999999999999},"max":{"1min":172.459,"5min":172.459,"15min":191.71299999999999},"last":13.978},{"interface":"front","average":{"1min":19.948,"5min":11.276999999999999,"15min":16.713000000000001},"min":{"1min":4.4370000000000003,"5min":4.4370000000000003,"15min":4.4370000000000003},"max":{"1min":174.608,"5min":174.608,"15min":244.46299999999999},"last":5.96}]},{"osd":3,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":19.504999999999999,"5min":15.193,"15min":20.762},"min":{"1min":10.653,"5min":8.2929999999999993,"15min":6.5830000000000002},"max":{"1min":70.066999999999993,"5min":70.066999999999993,"15min":223.97800000000001},"last":12.358000000000001},{"interface":"front","average":{"1min":18.291,"5min":14.847,"15min":22.268000000000001},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":6.8559999999999999},"max":{"1min":71.659999999999997,"5min":71.659999999999997,"15min":212.309},"last":16.094999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.582000000000001,"5min":14.68,"15min":19.289999999999999},"min":{"1min":10.567,"5min":8.952,"15min":7.0510000000000002},"max":{"1min":21.972000000000001,"5min":22.463000000000001,"15min":134.95699999999999},"last":17.088999999999999},{"interface":"front","average":{"1min":17.640999999999998,"5min":17.771000000000001,"15min":22.725000000000001},"min":{"1min":11.27,"5min":11.27,"15min":9.9809999999999999},"max":{"1min":22.663,"5min":22.663,"15min":135.798},"last":18.826000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":15.782,"5min":15.356,"15min":20.988},"min":{"1min":7.8040000000000003,"5min":7.8040000000000003,"15min":7.2869999999999999},"max":{"1min":20.332000000000001,"5min":20.966000000000001,"15min":288.577},"last":13.292999999999999},{"interface":"front","average":{"1min":17.925999999999998,"5min":17.777000000000001,"15min":23.446999999999999},"min":{"1min":12.861000000000001,"5min":12.124000000000001,"15min":10.234},"max":{"1min":23.948,"5min":23.948,"15min":142.77699999999999},"last":18.006}]},{"osd":7,"last update":"Fri Dec 1 17:35:48 2023","interfaces":[{"interface":"back","average":{"1min":11.468999999999999,"5min":10.596,"15min":15.210000000000001},"min":{"1min":7.2649999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"max":{"1min":21.209,"5min":21.209,"15min":168.59700000000001},"last":8.7439999999999998},{"interface":"front","average":{"1min":17.047000000000001,"5min":16.718,"15min":21.393000000000001},"min":{"1min":11.954000000000001,"5min":8.9459999999999997,"15min":8.4770000000000003},"max":{"1min":24.709,"5min":25.518000000000001,"15min":183.054},"last":15.342000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542536,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74332,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297508,"statfs":{"total":96636764160,"available":96560648192,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":24.530999999999999,"5min":11.712,"15min":15.721},"min":{"1min":4.3620000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"max":{"1min":201.333,"5min":201.333,"15min":228.261},"last":10.500999999999999},{"interface":"front","average":{"1min":27.792000000000002,"5min":13.209,"15min":17.690999999999999},"min":{"1min":6.5709999999999997,"5min":5.0339999999999998,"15min":4.6580000000000004},"max":{"1min":221.876,"5min":221.876,"15min":222.441},"last":16.646999999999998}]},{"osd":1,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.556000000000001,"5min":12.199999999999999,"15min":19.036000000000001},"min":{"1min":7.0490000000000004,"5min":6.0640000000000001,"15min":4.8899999999999997},"max":{"1min":66.040999999999997,"5min":66.040999999999997,"15min":216.20099999999999},"last":22.481999999999999},{"interface":"front","average":{"1min":17.574000000000002,"5min":12.794,"15min":19.777000000000001},"min":{"1min":7.7430000000000003,"5min":6.5650000000000004,"15min":5.6210000000000004},"max":{"1min":65.156999999999996,"5min":65.156999999999996,"15min":255.95599999999999},"last":23.436}]},{"osd":2,"last update":"Fri Dec 1 17:35:58 2023","interfaces":[{"interface":"back","average":{"1min":12.641999999999999,"5min":11.792,"15min":17.571999999999999},"min":{"1min":6.468,"5min":6.468,"15min":6.468},"max":{"1min":39.947000000000003,"5min":39.947000000000003,"15min":179.81999999999999},"last":12.893000000000001},{"interface":"front","average":{"1min":16.623999999999999,"5min":15.505000000000001,"15min":22.356999999999999},"min":{"1min":8.7070000000000007,"5min":8.7070000000000007,"15min":8.7070000000000007},"max":{"1min":42.759,"5min":42.759,"15min":224.47300000000001},"last":14.077999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":13.112,"15min":19.452999999999999},"min":{"1min":6.819,"5min":6.819,"15min":5.9020000000000001},"max":{"1min":61.933,"5min":61.933,"15min":248.07300000000001},"last":14.837999999999999},{"interface":"front","average":{"1min":22.122,"5min":16.693000000000001,"15min":23.396000000000001},"min":{"1min":11.053000000000001,"5min":10.221,"15min":9.7989999999999995},"max":{"1min":62.886000000000003,"5min":62.886000000000003,"15min":235.02699999999999},"last":19.724}]},{"osd":4,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":17.068999999999999,"5min":15.241,"15min":19.116},"min":{"1min":10.965999999999999,"5min":9.4930000000000003,"15min":8.1699999999999999},"max":{"1min":59.747,"5min":59.747,"15min":211.09399999999999},"last":17.584},{"interface":"front","average":{"1min":14.052,"5min":14.444000000000001,"15min":18.030000000000001},"min":{"1min":9.6010000000000009,"5min":8.6850000000000005,"15min":7.0540000000000003},"max":{"1min":29.402000000000001,"5min":29.402000000000001,"15min":216.85300000000001},"last":11.726000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":20.969999999999999,"5min":17.178000000000001,"15min":21.32},"min":{"1min":11.702,"5min":10.103,"15min":8.4339999999999993},"max":{"1min":111.396,"5min":111.396,"15min":223.89500000000001},"last":18.367000000000001},{"interface":"front","average":{"1min":16.765999999999998,"5min":14.01,"15min":14.154999999999999},"min":{"1min":7.5949999999999998,"5min":5.6829999999999998,"15min":5.6479999999999997},"max":{"1min":34.613,"5min":34.613,"15min":159.84399999999999},"last":12.284000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:35:55 2023","interfaces":[{"interface":"back","average":{"1min":18.885999999999999,"5min":17.652000000000001,"15min":21.867000000000001},"min":{"1min":10.467000000000001,"5min":10.467000000000001,"15min":9.6430000000000007},"max":{"1min":62.256999999999998,"5min":62.256999999999998,"15min":226.06100000000001},"last":13.481999999999999},{"interface":"front","average":{"1min":21.974,"5min":17.885000000000002,"15min":21.588999999999999},"min":{"1min":9.3369999999999997,"5min":9.3369999999999997,"15min":8.1820000000000004},"max":{"1min":125.431,"5min":125.431,"15min":317.267},"last":19.155000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542537,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74340,"kb_used_data":6860,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297500,"statfs":{"total":96636764160,"available":96560640000,"internally_reserved":0,"allocated":7024640,"data_stored":3846006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":7.8300000000000001,"5min":9.3949999999999996,"15min":16.763000000000002},"min":{"1min":4.1109999999999998,"5min":4.1109999999999998,"15min":3.7690000000000001},"max":{"1min":15.778,"5min":24.963999999999999,"15min":521.55899999999997},"last":10.879},{"interface":"front","average":{"1min":9.3529999999999998,"5min":9.8160000000000007,"15min":15.084},"min":{"1min":6.3360000000000003,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":13.864000000000001,"5min":21.491,"15min":281.125},"last":6.8289999999999997}]},{"osd":1,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":11.353999999999999,"5min":11.315,"15min":19.023},"min":{"1min":8.2140000000000004,"5min":6.7969999999999997,"15min":6.2770000000000001},"max":{"1min":14.935,"5min":21.539000000000001,"15min":308.99299999999999},"last":8.6709999999999994},{"interface":"front","average":{"1min":16.753,"5min":15.666,"15min":22.716000000000001},"min":{"1min":10.832000000000001,"5min":9.1069999999999993,"15min":6.3079999999999998},"max":{"1min":31.140000000000001,"5min":31.140000000000001,"15min":289.79399999999998},"last":19.161000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":16.849,"5min":15.061999999999999,"15min":21.503},"min":{"1min":11.24,"5min":9.1029999999999998,"15min":5.976},"max":{"1min":27.797999999999998,"5min":27.797999999999998,"15min":298.18200000000002},"last":16.841999999999999},{"interface":"front","average":{"1min":12.192,"5min":10.987,"15min":16.457999999999998},"min":{"1min":8.0640000000000001,"5min":6.4820000000000002,"15min":6.4820000000000002},"max":{"1min":25.771999999999998,"5min":25.771999999999998,"15min":223.768},"last":12.696999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":12.680999999999999,"15min":18.734000000000002},"min":{"1min":8.6850000000000005,"5min":7.1580000000000004,"15min":6.2169999999999996},"max":{"1min":26.462,"5min":26.462,"15min":243.40299999999999},"last":13.763999999999999},{"interface":"front","average":{"1min":17.468,"5min":16.356999999999999,"15min":22.004000000000001},"min":{"1min":13.864000000000001,"5min":10.156000000000001,"15min":8.4390000000000001},"max":{"1min":37.262999999999998,"5min":37.262999999999998,"15min":253.56100000000001},"last":14.471}]},{"osd":5,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":17.428999999999998,"5min":15.695,"15min":22.879999999999999},"min":{"1min":10.804,"5min":9.5199999999999996,"15min":6.6349999999999998},"max":{"1min":35.213999999999999,"5min":35.213999999999999,"15min":336.55599999999998},"last":15.127000000000001},{"interface":"front","average":{"1min":10.366,"5min":11.09,"15min":19.364000000000001},"min":{"1min":7.0019999999999998,"5min":6.4859999999999998,"15min":6.4790000000000001},"max":{"1min":14.853,"5min":23.786000000000001,"15min":357.50900000000001},"last":7.875}]},{"osd":6,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":14.206,"5min":15.972,"15min":20.510999999999999},"min":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":8.1080000000000005},"max":{"1min":34.234000000000002,"5min":34.234000000000002,"15min":255.52500000000001},"last":10.276999999999999},{"interface":"front","average":{"1min":19.222000000000001,"5min":15.115,"15min":19.218},"min":{"1min":15.247999999999999,"5min":7.7080000000000002,"15min":7.7080000000000002},"max":{"1min":37.884,"5min":37.884,"15min":312.791},"last":17.231999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":15.161,"5min":15.696,"15min":23.895},"min":{"1min":9.7590000000000003,"5min":9.6940000000000008,"15min":8.7330000000000005},"max":{"1min":23.195,"5min":23.195,"15min":413.48899999999998},"last":11.962},{"interface":"front","average":{"1min":17.57,"5min":16.561,"15min":21.308},"min":{"1min":9.4770000000000003,"5min":9.0809999999999995,"15min":6.6820000000000004},"max":{"1min":30.414999999999999,"5min":30.414999999999999,"15min":417.952},"last":17.997}]}]},{"osd":7,"up_from":13,"seq":55834575243,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69976,"kb_used_data":6668,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301864,"statfs":{"total":96636764160,"available":96565108736,"internally_reserved":0,"allocated":6828032,"data_stored":3717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":13.364000000000001,"5min":11.880000000000001,"15min":16.117000000000001},"min":{"1min":8.8580000000000005,"5min":5.2729999999999997,"15min":4.3360000000000003},"max":{"1min":39.423000000000002,"5min":39.423000000000002,"15min":203.19300000000001},"last":12.849},{"interface":"front","average":{"1min":17.373999999999999,"5min":15.071999999999999,"15min":18.241},"min":{"1min":11.513,"5min":6.4269999999999996,"15min":4.4199999999999999},"max":{"1min":40.698999999999998,"5min":40.698999999999998,"15min":204.13},"last":15.288}]},{"osd":1,"last update":"Fri Dec 1 17:35:57 2023","interfaces":[{"interface":"back","average":{"1min":7.9080000000000004,"5min":8.2289999999999992,"15min":15.226000000000001},"min":{"1min":6.1139999999999999,"5min":5.2300000000000004,"15min":5.0369999999999999},"max":{"1min":10.85,"5min":17.564,"15min":167.82900000000001},"last":6.093},{"interface":"front","average":{"1min":15.026,"5min":14.375999999999999,"15min":16.369},"min":{"1min":11.634,"5min":7.4870000000000001,"15min":5.484},"max":{"1min":22.280999999999999,"5min":22.280999999999999,"15min":154.137},"last":13.68}]},{"osd":2,"last update":"Fri Dec 1 17:36:23 2023","interfaces":[{"interface":"back","average":{"1min":11.753,"5min":10.013999999999999,"15min":16.608000000000001},"min":{"1min":5.8700000000000001,"5min":5.1820000000000004,"15min":5.1820000000000004},"max":{"1min":26.274999999999999,"5min":26.274999999999999,"15min":230.40600000000001},"last":10.253},{"interface":"front","average":{"1min":14.275,"5min":12.394,"15min":17.012},"min":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"max":{"1min":29.861000000000001,"5min":29.861000000000001,"15min":268.68000000000001},"last":6.8479999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":13.083,"15min":20.297000000000001},"min":{"1min":8.1359999999999992,"5min":6.5110000000000001,"15min":6.5110000000000001},"max":{"1min":85.724000000000004,"5min":85.724000000000004,"15min":290.15300000000002},"last":9.3230000000000004},{"interface":"front","average":{"1min":14.601000000000001,"5min":11.683,"15min":20.149000000000001},"min":{"1min":6.6319999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"max":{"1min":83.183000000000007,"5min":83.183000000000007,"15min":305.41399999999999},"last":8.5909999999999993}]},{"osd":4,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":14.577,"5min":14.051,"15min":18.489000000000001},"min":{"1min":10.007999999999999,"5min":8.5489999999999995,"15min":6.9299999999999997},"max":{"1min":17.609999999999999,"5min":22.071000000000002,"15min":149.93000000000001},"last":14.497},{"interface":"front","average":{"1min":17.727,"5min":17.503,"15min":23.488},"min":{"1min":14.632,"5min":11.496,"15min":9.7810000000000006},"max":{"1min":26.896999999999998,"5min":27.422000000000001,"15min":239.37},"last":16.16}]},{"osd":5,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":17.140000000000001,"5min":13.266,"15min":20.885000000000002},"min":{"1min":10.646000000000001,"5min":7.3209999999999997,"15min":6.8460000000000001},"max":{"1min":26.053999999999998,"5min":26.053999999999998,"15min":282.31200000000001},"last":16.951000000000001},{"interface":"front","average":{"1min":9.9749999999999996,"5min":13.909000000000001,"15min":19.908999999999999},"min":{"1min":6.8929999999999998,"5min":6.1529999999999996,"15min":6.1529999999999996},"max":{"1min":17.657,"5min":20.282,"15min":193.83600000000001},"last":7.7709999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:35:39 2023","interfaces":[{"interface":"back","average":{"1min":18.216999999999999,"5min":17.367999999999999,"15min":20.030999999999999},"min":{"1min":13.897,"5min":11.176,"15min":6.625},"max":{"1min":24.832999999999998,"5min":24.832999999999998,"15min":209.25},"last":17.841000000000001},{"interface":"front","average":{"1min":19.565999999999999,"5min":19.302,"15min":24.709},"min":{"1min":15.249000000000001,"5min":14.266999999999999,"15min":8.9420000000000002},"max":{"1min":27.905000000000001,"5min":27.905000000000001,"15min":193.184},"last":18.416}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:36:38.560 INFO:tasks.ceph:pgid 22.6 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.560 INFO:tasks.ceph:pgid 22.7 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.560 INFO:tasks.ceph:pgid 22.4 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.560 INFO:tasks.ceph:pgid 22.5 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.561 INFO:tasks.ceph:pgid 22.1a last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.561 INFO:tasks.ceph:pgid 22.1b last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.561 INFO:tasks.ceph:pgid 22.18 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.561 INFO:tasks.ceph:pgid 22.1f last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.561 INFO:tasks.ceph:pgid 22.1c last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.561 INFO:tasks.ceph:pgid 22.1d last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.562 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.562 INFO:tasks.ceph:pgid 22.17 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.562 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.562 INFO:tasks.ceph:pgid 22.14 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.562 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.562 INFO:tasks.ceph:pgid 22.15 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.563 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.563 INFO:tasks.ceph:pgid 22.16 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.563 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.563 INFO:tasks.ceph:pgid 22.10 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.563 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.563 INFO:tasks.ceph:pgid 22.11 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.563 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.12 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-01T17:03:59.560899+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=3, tm_sec=59, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.13 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.1 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.0 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.3 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.2 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.d last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.564 INFO:tasks.ceph:pgid 22.c last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.565 INFO:tasks.ceph:pgid 22.f last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.565 INFO:tasks.ceph:pgid 22.e last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.565 INFO:tasks.ceph:pgid 22.9 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.565 INFO:tasks.ceph:pgid 22.8 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.565 INFO:tasks.ceph:pgid 22.b last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.565 INFO:tasks.ceph:pgid 22.a last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:38.565 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-01T17:36:40.112 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:53.894 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.117 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:53.899 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.135 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:53.916 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.166 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:53.926 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.167 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:53.928 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.168 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:53.950 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.171 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:53.953 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.181 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:53.941 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.186 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:53.968 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.744 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:54.504 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.745 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:54.505 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.753 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:33:54.513 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.809 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:54.591 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.811 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:54.593 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.826 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:33:54.608 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.831 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:54.613 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.832 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:54.614 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:40.842 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:33:54.623 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.149 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:08.930 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.154 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:08.935 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.170 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:08.952 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.202 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:08.962 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.202 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:08.984 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.203 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:08.963 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.205 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:08.987 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.217 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:08.978 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:55.221 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:09.003 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:36:58.566 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:36:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.674+0000 7faa8b615640 10 monclient: get_monmap_and_config 2023-12-01T17:36:58.673 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.674+0000 7faa8b615640 10 monclient: build_initial_monmap 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.674+0000 7faa8b615640 10 monclient: monmap: 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:36:58.674961+0000 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:36:58.674961+0000 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:36:58.674 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.674+0000 7faa8b615640 1 Processor -- start 2023-12-01T17:36:58.676 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 1 -- start start 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient: init 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa840a8a20 addr v1:172.21.15.67:6789/0 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa840a8e00 0x7faa8413f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient(hunting): picked mon.noname-a con 0x7faa840a8e00 addr v2:172.21.15.67:3300/0 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faa841423b0 0x7faa84142840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient(hunting): picked mon.noname-c con 0x7faa841423b0 addr v2:172.21.15.92:3300/0 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa840b21f0 con 0x7faa840a8a20 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa840a9cc0 con 0x7faa840a8e00 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa840a9ed0 con 0x7faa841423b0 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:58.677 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.675+0000 7faa8b615640 10 monclient(hunting): authenticate will time out at 3349.913033s 2023-12-01T17:36:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa88bb0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7faa840a8e00 0x7faa8413f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa89bb2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faa841423b0 0x7faa84142840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa89bb2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7faa841423b0 0x7faa84142840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47020/0 (socket says 172.21.15.67:47020) 2023-12-01T17:36:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa89bb2640 1 -- 172.21.15.67:0/199602068 learned_addr learned my addr 172.21.15.67:0/199602068 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa89bb2640 10 monclient(hunting): get_auth_request con 0x7faa841423b0 auth_method 0 2023-12-01T17:36:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa89bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:58.678 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa89bb2640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa89bb2640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa88bb0640 10 monclient(hunting): get_auth_request con 0x7faa840a8e00 auth_method 0 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa88bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa88bb0640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.676+0000 7faa88bb0640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.677+0000 7faa89bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.677+0000 7faa89bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.677+0000 7faa89bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.677+0000 7faa88bb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.677+0000 7faa88bb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:58.682 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.677+0000 7faa88bb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 10 monclient(hunting): handle_auth_done global_id 23251 payload 274 2023-12-01T17:36:58.683 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 1 -- 172.21.15.67:0/199602068 >> v2:172.21.15.67:3300/0 conn(0x7faa840a8e00 msgr2=0x7faa8413f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 1 --2- 172.21.15.67:0/199602068 >> v2:172.21.15.67:3300/0 conn(0x7faa840a8e00 0x7faa8413f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 1 -- 172.21.15.67:0/199602068 >> v1:172.21.15.67:6789/0 conn(0x7faa840a8a20 legacy=0x7faa8413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 10 monclient: _finish_hunting 0 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 1 monclient: found mon.noname-c 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.678+0000 7faa89bb2640 1 -- 172.21.15.67:0/199602068 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa840aac70 con 0x7faa841423b0 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.680+0000 7faa89bb2640 1 --2- 172.21.15.67:0/199602068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84142840 secure :-1 s=READY pgs=5915 cs=0 l=1 rev1=1 crypto rx=0x7faa840b39a0 tx=0x7faa7c02b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e7ae16c90e20c8e server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 1 -- 172.21.15.67:0/199602068 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c0317e0 con 0x7faa841423b0 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: dump: 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:58.684 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: _finish_auth 0 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: _check_auth_tickets 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 1 -- 172.21.15.67:0/199602068 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c031940 con 0x7faa841423b0 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa8b615640 5 monclient: authenticate success, global_id 23251 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa8b615640 20 monclient: get_monmap_and_config waiting for monmap|config 2023-12-01T17:36:58.685 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 1 -- 172.21.15.67:0/199602068 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c031cd0 con 0x7faa841423b0 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa8b615640 10 monclient: get_monmap_and_config success 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa8b615640 4 set_mon_vals no callback set 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.684+0000 7faa7bfff640 10 monclient: dump: 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:58.686 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:58.687 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:58.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 10 monclient: shutdown 2023-12-01T17:36:58.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:58.689 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 -- 172.21.15.67:0/199602068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 msgr2=0x7faa84142840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 --2- 172.21.15.67:0/199602068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84142840 secure :-1 s=READY pgs=5915 cs=0 l=1 rev1=1 crypto rx=0x7faa840b39a0 tx=0x7faa7c02b880 comp rx=0 tx=0).stop 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 -- 172.21.15.67:0/199602068 shutdown_connections 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 --2- 172.21.15.67:0/199602068 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84142840 unknown :-1 s=CLOSED pgs=5915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 --2- 172.21.15.67:0/199602068 >> v2:172.21.15.67:3300/0 conn(0x7faa840a8e00 0x7faa8413f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 -- 172.21.15.67:0/199602068 >> 172.21.15.67:0/199602068 conn(0x7faa840b7870 msgr2=0x7faa840b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 -- 172.21.15.67:0/199602068 shutdown_connections 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 1 -- 172.21.15.67:0/199602068 wait complete. 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 10 monclient: build_initial_monmap 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.685+0000 7faa8b615640 10 monclient: monmap: 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:36:58.690 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 librados: starting msgr at 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 librados: starting objecter 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 Processor -- start 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 -- start start 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 librados: setting wanted keys 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 librados: calling monclient init 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 10 monclient: init 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 10 monclient: _reopen_session rank -1 2023-12-01T17:36:58.691 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 10 monclient: _add_conns ranks=[2,0,1] 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa840a8a20 0x7faa84137af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 10 monclient(hunting): picked mon.noname-c con 0x7faa840a8a20 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa840a8e00 0x7faa8413abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.686+0000 7faa8b615640 10 monclient(hunting): picked mon.noname-a con 0x7faa840a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84134950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 10 monclient(hunting): picked mon.noname-b con 0x7faa841423b0 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8413b630 con 0x7faa840a8e00 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa841438c0 con 0x7faa841423b0 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 10 monclient(hunting): start opening mon connection 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa84143a80 con 0x7faa840a8a20 2023-12-01T17:36:58.692 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 10 monclient(hunting): _renew_subs 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa8b615640 10 monclient(hunting): authenticate will time out at 3349.924433s 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa88bb0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa840a8e00 0x7faa8413abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa89bb2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84134950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa893b1640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa840a8a20 0x7faa84137af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa89bb2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84134950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:47022/0 (socket says 172.21.15.67:47022) 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa89bb2640 1 -- 172.21.15.67:0/266091093 learned_addr learned my addr 172.21.15.67:0/266091093 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa89bb2640 10 monclient(hunting): get_auth_request con 0x7faa841423b0 auth_method 0 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa89bb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa89bb2640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa89bb2640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa88bb0640 10 monclient(hunting): get_auth_request con 0x7faa840a8e00 auth_method 0 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa88bb0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa88bb0640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:58.693 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa88bb0640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa893b1640 10 monclient(hunting): get_auth_request con 0x7faa840a8a20 auth_method 0 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa893b1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa893b1640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.687+0000 7faa893b1640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa89bb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa89bb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa89bb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa88bb0640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa88bb0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa88bb0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa893b1640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa893b1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:36:58.694 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.688+0000 7faa893b1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:36:58.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.690+0000 7faa89bb2640 10 monclient(hunting): handle_auth_done global_id 23254 payload 824 2023-12-01T17:36:58.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.690+0000 7faa89bb2640 1 -- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa840a8a20 msgr2=0x7faa84137af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:58.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.690+0000 7faa89bb2640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa840a8a20 0x7faa84137af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:58.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.691+0000 7faa89bb2640 1 -- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa840a8e00 msgr2=0x7faa8413abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:58.695 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.691+0000 7faa89bb2640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa840a8e00 0x7faa8413abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.691+0000 7faa89bb2640 10 monclient: _finish_hunting 0 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.691+0000 7faa89bb2640 1 monclient: found mon.noname-b 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.691+0000 7faa89bb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.691+0000 7faa89bb2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.691+0000 7faa89bb2640 1 -- 172.21.15.67:0/266091093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c0094d0 con 0x7faa841423b0 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.693+0000 7faa89bb2640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84134950 secure :-1 s=READY pgs=5916 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0090b0 tx=0x7faa7c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.693+0000 7faa79ffb640 10 client.?.objecter ms_handle_connect 0x7faa841423b0 2023-12-01T17:36:58.696 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.693+0000 7faa79ffb640 10 client.?.objecter resend_mon_ops 2023-12-01T17:36:58.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 1 -- 172.21.15.67:0/266091093 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c031f10 con 0x7faa841423b0 2023-12-01T17:36:58.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:58.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:36:58.697 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: dump: 2023-12-01T17:36:58.697 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:58.697 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: _finish_auth 0 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: _check_auth_tickets 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 1 -- 172.21.15.67:0/266091093 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7c032880 con 0x7faa841423b0 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:36:58.698 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 1 -- 172.21.15.67:0/266091093 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c032c10 con 0x7faa841423b0 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa79ffb640 10 monclient: dump: 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa8b615640 5 monclient: authenticate success, global_id 23254 2023-12-01T17:36:58.699 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa8b615640 10 monclient: _renew_subs 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa8b615640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8413b110 con 0x7faa841423b0 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.696+0000 7faa7b7fe640 4 set_mon_vals no callback set 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.697+0000 7faa8b615640 10 client.23254.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.697+0000 7faa8b615640 10 monclient: _renew_subs 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.697+0000 7faa8b615640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.697+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa84134e90 con 0x7faa841423b0 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.697+0000 7faa8b615640 1 librados: init done 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.698+0000 7faa8b615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.698+0000 7faa8b615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.698+0000 7faa8b615640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:36:58.700 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.698+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8404e840 con 0x7faa841423b0 2023-12-01T17:36:58.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.717+0000 7faa79ffb640 1 -- 172.21.15.67:0/266091093 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7faa7c04d3f0 con 0x7faa841423b0 2023-12-01T17:36:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.717+0000 7faa79ffb640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7faa6806c870 0x7faa6806ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:36:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.717+0000 7faa893b1640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7faa6806c870 0x7faa6806ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:36:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.717+0000 7faa893b1640 10 monclient: get_auth_request con 0x7faa6806c870 auth_method 0 2023-12-01T17:36:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.718+0000 7faa79ffb640 10 client.23254.objecter ms_handle_connect 0x7faa6806c870 2023-12-01T17:36:58.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.718+0000 7faa893b1640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7faa6806c870 0x7faa6806ed30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7faa80001e70 tx=0x7faa80009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:36:58.934 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.935+0000 7faa79ffb640 1 -- 172.21.15.67:0/266091093 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7faa7c07a140 con 0x7faa841423b0 2023-12-01T17:36:58.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.935+0000 7faa79ffb640 10 client.23254.objecter ms_dispatch 0x7faa84000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:36:58.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.935+0000 7faa79ffb640 3 client.23254.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:36:58.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.935+0000 7faa79ffb640 3 client.23254.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:36:58.935 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.935+0000 7faa79ffb640 20 client.23254.objecter dump_active .. 0 homeless 2023-12-01T17:36:58.944 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.944+0000 7faa79ffb640 1 -- 172.21.15.67:0/266091093 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa7c076480 con 0x7faa841423b0 2023-12-01T17:36:58.944 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.944+0000 7faa79ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:36:58.944 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:58.944+0000 7faa79ffb640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:36:59.117 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.117+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faa8413b7a0 con 0x7faa6806c870 2023-12-01T17:36:59.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.122+0000 7faa79ffb640 1 -- 172.21.15.67:0/266091093 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152077 (secure 0 0 0) 0x7faa8413b7a0 con 0x7faa6806c870 2023-12-01T17:36:59.123 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:36:59.124 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:36:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 10 librados: watch_flush enter 2023-12-01T17:36:59.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 10 librados: watch_flush exit 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 20 client.23254.objecter shutdown clearing up homeless session... 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 10 client.23254.objecter successfully canceled tick 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7faa6806c870 msgr2=0x7faa6806ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7faa6806c870 0x7faa6806ed30 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7faa80001e70 tx=0x7faa80009040 comp rx=0 tx=0).stop 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 10 monclient: shutdown 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 msgr2=0x7faa84134950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84134950 secure :-1 s=READY pgs=5916 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0090b0 tx=0x7faa7c0046c0 comp rx=0 tx=0).stop 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 shutdown_connections 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.124+0000 7faa8b615640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7faa841423b0 0x7faa84134950 unknown :-1 s=CLOSED pgs=5916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.125+0000 7faa8b615640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa840a8e00 0x7faa8413abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.125+0000 7faa8b615640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7faa6806c870 0x7faa6806ed30 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.125+0000 7faa8b615640 1 --2- 172.21.15.67:0/266091093 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7faa840a8a20 0x7faa84137af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:36:59.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.125+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 >> 172.21.15.67:0/266091093 conn(0x7faa840b7870 msgr2=0x7faa840b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:36:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.125+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 shutdown_connections 2023-12-01T17:36:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.125+0000 7faa8b615640 1 -- 172.21.15.67:0/266091093 wait complete. 2023-12-01T17:36:59.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:36:59.125+0000 7faa8b615640 1 librados: shutdown 2023-12-01T17:36:59.135 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":2096,"stamp":"2023-12-01T17:36:58.780455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":567196,"kb_used_data":53980,"kb_used_omap":12,"kb_used_meta":513139,"kb_avail":754407524,"statfs":{"total":773094113280,"available":772513304576,"internally_reserved":0,"allocated":55275520,"data_stored":30438252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":525454896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002526"},"pg_stats":[{"pgid":"22.6","version":"0'0","reported_seq":1050,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:27.500215+0000","last_change":"2023-12-01T17:05:14.287690+0000","last_active":"2023-12-01T17:36:27.500215+0000","last_peered":"2023-12-01T17:36:27.500215+0000","last_clean":"2023-12-01T17:36:27.500215+0000","last_became_active":"2023-12-01T17:05:14.257152+0000","last_became_peered":"2023-12-01T17:05:14.257152+0000","last_unstale":"2023-12-01T17:36:27.500215+0000","last_undegraded":"2023-12-01T17:36:27.500215+0000","last_fullsized":"2023-12-01T17:36:27.500215+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:05:03.392191+0000","last_clean_scrub_stamp":"2023-12-01T17:05:03.392191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1066,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:53.343956+0000","last_change":"2023-12-01T17:36:53.344682+0000","last_active":"2023-12-01T17:36:53.343956+0000","last_peered":"2023-12-01T17:36:53.343956+0000","last_clean":"2023-12-01T17:36:53.343956+0000","last_became_active":"2023-12-01T17:05:21.296791+0000","last_became_peered":"2023-12-01T17:05:21.296791+0000","last_unstale":"2023-12-01T17:36:53.343956+0000","last_undegraded":"2023-12-01T17:36:53.343956+0000","last_fullsized":"2023-12-01T17:36:53.343956+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:53.342520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:53.342520+0000","last_clean_scrub_stamp":"2023-12-01T17:36:53.342520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:54:08.559436+0000","scrub_duration":0.079442730000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1066,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:45.166465+0000","last_change":"2023-12-01T17:36:45.167142+0000","last_active":"2023-12-01T17:36:45.166465+0000","last_peered":"2023-12-01T17:36:45.166465+0000","last_clean":"2023-12-01T17:36:45.166465+0000","last_became_active":"2023-12-01T17:05:13.265267+0000","last_became_peered":"2023-12-01T17:05:13.265267+0000","last_unstale":"2023-12-01T17:36:45.166465+0000","last_undegraded":"2023-12-01T17:36:45.166465+0000","last_fullsized":"2023-12-01T17:36:45.166465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:45.165081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:45.165081+0000","last_clean_scrub_stamp":"2023-12-01T17:36:45.165081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:07:18.537211+0000","scrub_duration":0.027418776999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.953859+0000","last_change":"2023-12-01T17:36:44.954614+0000","last_active":"2023-12-01T17:36:44.953859+0000","last_peered":"2023-12-01T17:36:44.953859+0000","last_clean":"2023-12-01T17:36:44.953859+0000","last_became_active":"2023-12-01T17:05:38.651540+0000","last_became_peered":"2023-12-01T17:05:38.651540+0000","last_unstale":"2023-12-01T17:36:44.953859+0000","last_undegraded":"2023-12-01T17:36:44.953859+0000","last_fullsized":"2023-12-01T17:36:44.953859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.952389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.952389+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.952389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:51:40.516240+0000","scrub_duration":0.077883641000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1071,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:40.853151+0000","last_change":"2023-12-01T17:36:40.853841+0000","last_active":"2023-12-01T17:36:40.853151+0000","last_peered":"2023-12-01T17:36:40.853151+0000","last_clean":"2023-12-01T17:36:40.853151+0000","last_became_active":"2023-12-01T17:05:45.102496+0000","last_became_peered":"2023-12-01T17:05:45.102496+0000","last_unstale":"2023-12-01T17:36:40.853151+0000","last_undegraded":"2023-12-01T17:36:40.853151+0000","last_fullsized":"2023-12-01T17:36:40.853151+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:40.851788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:40.851788+0000","last_clean_scrub_stamp":"2023-12-01T17:36:40.851788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:49:20.220515+0000","scrub_duration":0.042120170999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1061,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:35.965140+0000","last_change":"2023-12-01T17:36:35.966071+0000","last_active":"2023-12-01T17:36:35.965140+0000","last_peered":"2023-12-01T17:36:35.965140+0000","last_clean":"2023-12-01T17:36:35.965140+0000","last_became_active":"2023-12-01T17:05:21.112377+0000","last_became_peered":"2023-12-01T17:05:21.112377+0000","last_unstale":"2023-12-01T17:36:35.965140+0000","last_undegraded":"2023-12-01T17:36:35.965140+0000","last_fullsized":"2023-12-01T17:36:35.965140+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:35.963469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:35.963469+0000","last_clean_scrub_stamp":"2023-12-01T17:36:35.963469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:36:03.057210+0000","scrub_duration":0.081911166999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:41.123404+0000","last_change":"2023-12-01T17:36:41.123404+0000","last_active":"2023-12-01T17:36:41.123404+0000","last_peered":"2023-12-01T17:36:41.123404+0000","last_clean":"2023-12-01T17:36:41.123404+0000","last_became_active":"2023-12-01T17:05:12.325443+0000","last_became_peered":"2023-12-01T17:05:12.325443+0000","last_unstale":"2023-12-01T17:36:41.123404+0000","last_undegraded":"2023-12-01T17:36:41.123404+0000","last_fullsized":"2023-12-01T17:36:41.123404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:41.121365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:41.121365+0000","last_clean_scrub_stamp":"2023-12-01T17:36:41.121365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:25:47.490775+0000","scrub_duration":0.031595139000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1062,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:33.940179+0000","last_change":"2023-12-01T17:36:33.940179+0000","last_active":"2023-12-01T17:36:33.940179+0000","last_peered":"2023-12-01T17:36:33.940179+0000","last_clean":"2023-12-01T17:36:33.940179+0000","last_became_active":"2023-12-01T17:05:35.971286+0000","last_became_peered":"2023-12-01T17:05:35.971286+0000","last_unstale":"2023-12-01T17:36:33.940179+0000","last_undegraded":"2023-12-01T17:36:33.940179+0000","last_fullsized":"2023-12-01T17:36:33.940179+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:33.933861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:33.933861+0000","last_clean_scrub_stamp":"2023-12-01T17:36:33.933861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:29:13.183460+0000","scrub_duration":0.050039922000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1057,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:32.519091+0000","last_change":"2023-12-01T17:36:32.519854+0000","last_active":"2023-12-01T17:36:32.519091+0000","last_peered":"2023-12-01T17:36:32.519091+0000","last_clean":"2023-12-01T17:36:32.519091+0000","last_became_active":"2023-12-01T17:05:33.448448+0000","last_became_peered":"2023-12-01T17:05:33.448448+0000","last_unstale":"2023-12-01T17:36:32.519091+0000","last_undegraded":"2023-12-01T17:36:32.519091+0000","last_fullsized":"2023-12-01T17:36:32.519091+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:32.517436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:32.517436+0000","last_clean_scrub_stamp":"2023-12-01T17:36:32.517436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:57:35.099944+0000","scrub_duration":0.069663173999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:43.961410+0000","last_change":"2023-12-01T17:36:43.962127+0000","last_active":"2023-12-01T17:36:43.961410+0000","last_peered":"2023-12-01T17:36:43.961410+0000","last_clean":"2023-12-01T17:36:43.961410+0000","last_became_active":"2023-12-01T17:05:36.329117+0000","last_became_peered":"2023-12-01T17:05:36.329117+0000","last_unstale":"2023-12-01T17:36:43.961410+0000","last_undegraded":"2023-12-01T17:36:43.961410+0000","last_fullsized":"2023-12-01T17:36:43.961410+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:43.959988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:43.959988+0000","last_clean_scrub_stamp":"2023-12-01T17:36:43.959988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:13:24.416511+0000","scrub_duration":0.068376606000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.136422+0000","last_change":"2023-12-01T17:36:44.137094+0000","last_active":"2023-12-01T17:36:44.136422+0000","last_peered":"2023-12-01T17:36:44.136422+0000","last_clean":"2023-12-01T17:36:44.136422+0000","last_became_active":"2023-12-01T17:05:11.452081+0000","last_became_peered":"2023-12-01T17:05:11.452081+0000","last_unstale":"2023-12-01T17:36:44.136422+0000","last_undegraded":"2023-12-01T17:36:44.136422+0000","last_fullsized":"2023-12-01T17:36:44.136422+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.135069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.135069+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.135069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:37:10.964902+0000","scrub_duration":0.027982248000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:47.339466+0000","last_change":"2023-12-01T17:36:47.340125+0000","last_active":"2023-12-01T17:36:47.339466+0000","last_peered":"2023-12-01T17:36:47.339466+0000","last_clean":"2023-12-01T17:36:47.339466+0000","last_became_active":"2023-12-01T17:05:36.185378+0000","last_became_peered":"2023-12-01T17:05:36.185378+0000","last_unstale":"2023-12-01T17:36:47.339466+0000","last_undegraded":"2023-12-01T17:36:47.339466+0000","last_fullsized":"2023-12-01T17:36:47.339466+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:47.338105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:47.338105+0000","last_clean_scrub_stamp":"2023-12-01T17:36:47.338105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:50:41.146788+0000","scrub_duration":0.064980219000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1099'78","reported_seq":1174,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:34.048890+0000","last_change":"2023-12-01T17:36:34.050160+0000","last_active":"2023-12-01T17:36:34.048890+0000","last_peered":"2023-12-01T17:36:34.048890+0000","last_clean":"2023-12-01T17:36:34.048890+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:36:34.048890+0000","last_undegraded":"2023-12-01T17:36:34.048890+0000","last_fullsized":"2023-12-01T17:36:34.048890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'78","last_scrub_stamp":"2023-12-01T17:36:34.044757+0000","last_deep_scrub":"1099'78","last_deep_scrub_stamp":"2023-12-01T17:36:34.044757+0000","last_clean_scrub_stamp":"2023-12-01T17:36:34.044757+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:16:14.393778+0000","scrub_duration":0.143069317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1062,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:38.933961+0000","last_change":"2023-12-01T17:36:38.934686+0000","last_active":"2023-12-01T17:36:38.933961+0000","last_peered":"2023-12-01T17:36:38.933961+0000","last_clean":"2023-12-01T17:36:38.933961+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:36:38.933961+0000","last_undegraded":"2023-12-01T17:36:38.933961+0000","last_fullsized":"2023-12-01T17:36:38.933961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:38.932452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:38.932452+0000","last_clean_scrub_stamp":"2023-12-01T17:36:38.932452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:21:52.995129+0000","scrub_duration":0.044391881000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1058,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:46.149295+0000","last_change":"2023-12-01T17:36:46.150030+0000","last_active":"2023-12-01T17:36:46.149295+0000","last_peered":"2023-12-01T17:36:46.149295+0000","last_clean":"2023-12-01T17:36:46.149295+0000","last_became_active":"2023-12-01T17:05:15.659631+0000","last_became_peered":"2023-12-01T17:05:15.659631+0000","last_unstale":"2023-12-01T17:36:46.149295+0000","last_undegraded":"2023-12-01T17:36:46.149295+0000","last_fullsized":"2023-12-01T17:36:46.149295+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:46.147862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:46.147862+0000","last_clean_scrub_stamp":"2023-12-01T17:36:46.147862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T04:49:27.130671+0000","scrub_duration":0.076369635000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:35.850307+0000","last_change":"2023-12-01T17:36:35.851277+0000","last_active":"2023-12-01T17:36:35.850307+0000","last_peered":"2023-12-01T17:36:35.850307+0000","last_clean":"2023-12-01T17:36:35.850307+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:36:35.850307+0000","last_undegraded":"2023-12-01T17:36:35.850307+0000","last_fullsized":"2023-12-01T17:36:35.850307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:35.848834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:35.848834+0000","last_clean_scrub_stamp":"2023-12-01T17:36:35.848834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:13:37.157785+0000","scrub_duration":0.034576843000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:39.047159+0000","last_change":"2023-12-01T17:36:39.047877+0000","last_active":"2023-12-01T17:36:39.047159+0000","last_peered":"2023-12-01T17:36:39.047159+0000","last_clean":"2023-12-01T17:36:39.047159+0000","last_became_active":"2023-12-01T17:05:11.329382+0000","last_became_peered":"2023-12-01T17:05:11.329382+0000","last_unstale":"2023-12-01T17:36:39.047159+0000","last_undegraded":"2023-12-01T17:36:39.047159+0000","last_fullsized":"2023-12-01T17:36:39.047159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:39.045715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:39.045715+0000","last_clean_scrub_stamp":"2023-12-01T17:36:39.045715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:19:56.059217+0000","scrub_duration":0.032303137000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1070,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.338016+0000","last_change":"2023-12-01T17:36:44.338677+0000","last_active":"2023-12-01T17:36:44.338016+0000","last_peered":"2023-12-01T17:36:44.338016+0000","last_clean":"2023-12-01T17:36:44.338016+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:36:44.338016+0000","last_undegraded":"2023-12-01T17:36:44.338016+0000","last_fullsized":"2023-12-01T17:36:44.338016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.336665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.336665+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.336665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:01:19.263355+0000","scrub_duration":0.040984052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:49.138747+0000","last_change":"2023-12-01T17:36:49.139443+0000","last_active":"2023-12-01T17:36:49.138747+0000","last_peered":"2023-12-01T17:36:49.138747+0000","last_clean":"2023-12-01T17:36:49.138747+0000","last_became_active":"2023-12-01T17:05:16.481672+0000","last_became_peered":"2023-12-01T17:05:16.481672+0000","last_unstale":"2023-12-01T17:36:49.138747+0000","last_undegraded":"2023-12-01T17:36:49.138747+0000","last_fullsized":"2023-12-01T17:36:49.138747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:49.137364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:49.137364+0000","last_clean_scrub_stamp":"2023-12-01T17:36:49.137364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:28:04.767796+0000","scrub_duration":0.050542482999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:43.972358+0000","last_change":"2023-12-01T17:36:43.973083+0000","last_active":"2023-12-01T17:36:43.972358+0000","last_peered":"2023-12-01T17:36:43.972358+0000","last_clean":"2023-12-01T17:36:43.972358+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:36:43.972358+0000","last_undegraded":"2023-12-01T17:36:43.972358+0000","last_fullsized":"2023-12-01T17:36:43.972358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:43.970938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:43.970938+0000","last_clean_scrub_stamp":"2023-12-01T17:36:43.970938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:30:06.736333+0000","scrub_duration":0.035023907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:50.267325+0000","last_change":"2023-12-01T17:36:50.267970+0000","last_active":"2023-12-01T17:36:50.267325+0000","last_peered":"2023-12-01T17:36:50.267325+0000","last_clean":"2023-12-01T17:36:50.267325+0000","last_became_active":"2023-12-01T17:05:14.948506+0000","last_became_peered":"2023-12-01T17:05:14.948506+0000","last_unstale":"2023-12-01T17:36:50.267325+0000","last_undegraded":"2023-12-01T17:36:50.267325+0000","last_fullsized":"2023-12-01T17:36:50.267325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:50.265821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:50.265821+0000","last_clean_scrub_stamp":"2023-12-01T17:36:50.265821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:30:58.709148+0000","scrub_duration":0.039070454999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1069,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:37.179004+0000","last_change":"2023-12-01T17:36:37.179968+0000","last_active":"2023-12-01T17:36:37.179004+0000","last_peered":"2023-12-01T17:36:37.179004+0000","last_clean":"2023-12-01T17:36:37.179004+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:36:37.179004+0000","last_undegraded":"2023-12-01T17:36:37.179004+0000","last_fullsized":"2023-12-01T17:36:37.179004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:37.177549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:37.177549+0000","last_clean_scrub_stamp":"2023-12-01T17:36:37.177549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:55:33.275441+0000","scrub_duration":0.035436190999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:37.013581+0000","last_change":"2023-12-01T17:36:37.014317+0000","last_active":"2023-12-01T17:36:37.013581+0000","last_peered":"2023-12-01T17:36:37.013581+0000","last_clean":"2023-12-01T17:36:37.013581+0000","last_became_active":"2023-12-01T17:05:11.693016+0000","last_became_peered":"2023-12-01T17:05:11.693016+0000","last_unstale":"2023-12-01T17:36:37.013581+0000","last_undegraded":"2023-12-01T17:36:37.013581+0000","last_fullsized":"2023-12-01T17:36:37.013581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:37.012131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:37.012131+0000","last_clean_scrub_stamp":"2023-12-01T17:36:37.012131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:18:23.046760+0000","scrub_duration":0.033385561000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:49.902910+0000","last_change":"2023-12-01T17:36:49.903621+0000","last_active":"2023-12-01T17:36:49.902910+0000","last_peered":"2023-12-01T17:36:49.902910+0000","last_clean":"2023-12-01T17:36:49.902910+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:36:49.902910+0000","last_undegraded":"2023-12-01T17:36:49.902910+0000","last_fullsized":"2023-12-01T17:36:49.902910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:49.901478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:49.901478+0000","last_clean_scrub_stamp":"2023-12-01T17:36:49.901478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:09:38.122593+0000","scrub_duration":0.037209064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1075,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:42.514285+0000","last_change":"2023-12-01T17:36:42.514948+0000","last_active":"2023-12-01T17:36:42.514285+0000","last_peered":"2023-12-01T17:36:42.514285+0000","last_clean":"2023-12-01T17:36:42.514285+0000","last_became_active":"2023-12-01T17:05:12.768446+0000","last_became_peered":"2023-12-01T17:05:12.768446+0000","last_unstale":"2023-12-01T17:36:42.514285+0000","last_undegraded":"2023-12-01T17:36:42.514285+0000","last_fullsized":"2023-12-01T17:36:42.514285+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:42.512983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:42.512983+0000","last_clean_scrub_stamp":"2023-12-01T17:36:42.512983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:54:53.202156+0000","scrub_duration":0.051052442000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1077,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:37.836360+0000","last_change":"2023-12-01T17:36:37.837000+0000","last_active":"2023-12-01T17:36:37.836360+0000","last_peered":"2023-12-01T17:36:37.836360+0000","last_clean":"2023-12-01T17:36:37.836360+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:36:37.836360+0000","last_undegraded":"2023-12-01T17:36:37.836360+0000","last_fullsized":"2023-12-01T17:36:37.836360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:37.835002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:37.835002+0000","last_clean_scrub_stamp":"2023-12-01T17:36:37.835002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:52:23.931657+0000","scrub_duration":0.033273367999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1066,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:45.341446+0000","last_change":"2023-12-01T17:36:45.342107+0000","last_active":"2023-12-01T17:36:45.341446+0000","last_peered":"2023-12-01T17:36:45.341446+0000","last_clean":"2023-12-01T17:36:45.341446+0000","last_became_active":"2023-12-01T17:05:24.924547+0000","last_became_peered":"2023-12-01T17:05:24.924547+0000","last_unstale":"2023-12-01T17:36:45.341446+0000","last_undegraded":"2023-12-01T17:36:45.341446+0000","last_fullsized":"2023-12-01T17:36:45.341446+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:45.340092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:45.340092+0000","last_clean_scrub_stamp":"2023-12-01T17:36:45.340092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:42:17.355162+0000","scrub_duration":0.034252206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.952033+0000","last_change":"2023-12-01T17:36:44.952769+0000","last_active":"2023-12-01T17:36:44.952033+0000","last_peered":"2023-12-01T17:36:44.952033+0000","last_clean":"2023-12-01T17:36:44.952033+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:36:44.952033+0000","last_undegraded":"2023-12-01T17:36:44.952033+0000","last_fullsized":"2023-12-01T17:36:44.952033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.948272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.948272+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.948272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:34:41.622283+0000","scrub_duration":0.049061754999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1072,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:38.895535+0000","last_change":"2023-12-01T17:36:38.896418+0000","last_active":"2023-12-01T17:36:38.895535+0000","last_peered":"2023-12-01T17:36:38.895535+0000","last_clean":"2023-12-01T17:36:38.895535+0000","last_became_active":"2023-12-01T17:05:28.748541+0000","last_became_peered":"2023-12-01T17:05:28.748541+0000","last_unstale":"2023-12-01T17:36:38.895535+0000","last_undegraded":"2023-12-01T17:36:38.895535+0000","last_fullsized":"2023-12-01T17:36:38.895535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:38.894158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:38.894158+0000","last_clean_scrub_stamp":"2023-12-01T17:36:38.894158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:49:49.640669+0000","scrub_duration":0.076043471000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1","version":"1099'8","reported_seq":1078,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:47.847022+0000","last_change":"2023-12-01T17:36:47.848174+0000","last_active":"2023-12-01T17:36:47.847022+0000","last_peered":"2023-12-01T17:36:47.847022+0000","last_clean":"2023-12-01T17:36:47.847022+0000","last_became_active":"2023-12-01T17:05:38.775069+0000","last_became_peered":"2023-12-01T17:05:38.775069+0000","last_unstale":"2023-12-01T17:36:47.847022+0000","last_undegraded":"2023-12-01T17:36:47.847022+0000","last_fullsized":"2023-12-01T17:36:47.847022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'8","last_scrub_stamp":"2023-12-01T17:36:47.845650+0000","last_deep_scrub":"1099'8","last_deep_scrub_stamp":"2023-12-01T17:36:47.845650+0000","last_clean_scrub_stamp":"2023-12-01T17:36:47.845650+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:06:25.294586+0000","scrub_duration":0.084520799999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1065,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:34.895082+0000","last_change":"2023-12-01T17:36:34.895800+0000","last_active":"2023-12-01T17:36:34.895082+0000","last_peered":"2023-12-01T17:36:34.895082+0000","last_clean":"2023-12-01T17:36:34.895082+0000","last_became_active":"2023-12-01T17:05:17.447969+0000","last_became_peered":"2023-12-01T17:05:17.447969+0000","last_unstale":"2023-12-01T17:36:34.895082+0000","last_undegraded":"2023-12-01T17:36:34.895082+0000","last_fullsized":"2023-12-01T17:36:34.895082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:34.893658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:34.893658+0000","last_clean_scrub_stamp":"2023-12-01T17:36:34.893658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:27:39.596435+0000","scrub_duration":0.034514954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1062,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:38.080186+0000","last_change":"2023-12-01T17:36:38.080912+0000","last_active":"2023-12-01T17:36:38.080186+0000","last_peered":"2023-12-01T17:36:38.080186+0000","last_clean":"2023-12-01T17:36:38.080186+0000","last_became_active":"2023-12-01T17:05:12.334114+0000","last_became_peered":"2023-12-01T17:05:12.334114+0000","last_unstale":"2023-12-01T17:36:38.080186+0000","last_undegraded":"2023-12-01T17:36:38.080186+0000","last_fullsized":"2023-12-01T17:36:38.080186+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:38.076259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:38.076259+0000","last_clean_scrub_stamp":"2023-12-01T17:36:38.076259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:35:13.808207+0000","scrub_duration":0.069810686999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1077,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:43.482858+0000","last_change":"2023-12-01T17:36:43.483544+0000","last_active":"2023-12-01T17:36:43.482858+0000","last_peered":"2023-12-01T17:36:43.482858+0000","last_clean":"2023-12-01T17:36:43.482858+0000","last_became_active":"2023-12-01T17:05:39.340484+0000","last_became_peered":"2023-12-01T17:05:39.340484+0000","last_unstale":"2023-12-01T17:36:43.482858+0000","last_undegraded":"2023-12-01T17:36:43.482858+0000","last_fullsized":"2023-12-01T17:36:43.482858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:43.481522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:43.481522+0000","last_clean_scrub_stamp":"2023-12-01T17:36:43.481522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:00:59.719215+0000","scrub_duration":0.034362386000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:40.021477+0000","last_change":"2023-12-01T17:36:40.021477+0000","last_active":"2023-12-01T17:36:40.021477+0000","last_peered":"2023-12-01T17:36:40.021477+0000","last_clean":"2023-12-01T17:36:40.021477+0000","last_became_active":"2023-12-01T17:05:58.772230+0000","last_became_peered":"2023-12-01T17:05:58.772230+0000","last_unstale":"2023-12-01T17:36:40.021477+0000","last_undegraded":"2023-12-01T17:36:40.021477+0000","last_fullsized":"2023-12-01T17:36:40.021477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:40.019346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:40.019346+0000","last_clean_scrub_stamp":"2023-12-01T17:36:40.019346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:34:11.691955+0000","scrub_duration":0.049893717999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1058,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:51.958138+0000","last_change":"2023-12-01T17:36:51.958865+0000","last_active":"2023-12-01T17:36:51.958138+0000","last_peered":"2023-12-01T17:36:51.958138+0000","last_clean":"2023-12-01T17:36:51.958138+0000","last_became_active":"2023-12-01T17:05:11.209283+0000","last_became_peered":"2023-12-01T17:05:11.209283+0000","last_unstale":"2023-12-01T17:36:51.958138+0000","last_undegraded":"2023-12-01T17:36:51.958138+0000","last_fullsized":"2023-12-01T17:36:51.958138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:51.954556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:51.954556+0000","last_clean_scrub_stamp":"2023-12-01T17:36:51.954556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:15:04.690916+0000","scrub_duration":0.046634676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1059,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:45.121735+0000","last_change":"2023-12-01T17:36:45.122583+0000","last_active":"2023-12-01T17:36:45.121735+0000","last_peered":"2023-12-01T17:36:45.121735+0000","last_clean":"2023-12-01T17:36:45.121735+0000","last_became_active":"2023-12-01T17:05:17.215461+0000","last_became_peered":"2023-12-01T17:05:17.215461+0000","last_unstale":"2023-12-01T17:36:45.121735+0000","last_undegraded":"2023-12-01T17:36:45.121735+0000","last_fullsized":"2023-12-01T17:36:45.121735+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:45.119993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:45.119993+0000","last_clean_scrub_stamp":"2023-12-01T17:36:45.119993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:37:06.008536+0000","scrub_duration":0.094950128999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1056,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:36.547455+0000","last_change":"2023-12-01T17:36:36.547455+0000","last_active":"2023-12-01T17:36:36.547455+0000","last_peered":"2023-12-01T17:36:36.547455+0000","last_clean":"2023-12-01T17:36:36.547455+0000","last_became_active":"2023-12-01T17:05:21.668214+0000","last_became_peered":"2023-12-01T17:05:21.668214+0000","last_unstale":"2023-12-01T17:36:36.547455+0000","last_undegraded":"2023-12-01T17:36:36.547455+0000","last_fullsized":"2023-12-01T17:36:36.547455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:36.542404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:36.542404+0000","last_clean_scrub_stamp":"2023-12-01T17:36:36.542404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:29:17.880198+0000","scrub_duration":0.041798604000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1073,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:42.788666+0000","last_change":"2023-12-01T17:36:42.789790+0000","last_active":"2023-12-01T17:36:42.788666+0000","last_peered":"2023-12-01T17:36:42.788666+0000","last_clean":"2023-12-01T17:36:42.788666+0000","last_became_active":"2023-12-01T17:05:46.016610+0000","last_became_peered":"2023-12-01T17:05:46.016610+0000","last_unstale":"2023-12-01T17:36:42.788666+0000","last_undegraded":"2023-12-01T17:36:42.788666+0000","last_fullsized":"2023-12-01T17:36:42.788666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:42.786419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:42.786419+0000","last_clean_scrub_stamp":"2023-12-01T17:36:42.786419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:14:29.992685+0000","scrub_duration":0.054486035000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1059,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:50.899559+0000","last_change":"2023-12-01T17:36:50.900297+0000","last_active":"2023-12-01T17:36:50.899559+0000","last_peered":"2023-12-01T17:36:50.899559+0000","last_clean":"2023-12-01T17:36:50.899559+0000","last_became_active":"2023-12-01T17:05:37.358444+0000","last_became_peered":"2023-12-01T17:05:37.358444+0000","last_unstale":"2023-12-01T17:36:50.899559+0000","last_undegraded":"2023-12-01T17:36:50.899559+0000","last_fullsized":"2023-12-01T17:36:50.899559+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:50.895912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:50.895912+0000","last_clean_scrub_stamp":"2023-12-01T17:36:50.895912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:10:33.121163+0000","scrub_duration":0.031742754999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1065,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:48.133542+0000","last_change":"2023-12-01T17:36:48.134215+0000","last_active":"2023-12-01T17:36:48.133542+0000","last_peered":"2023-12-01T17:36:48.133542+0000","last_clean":"2023-12-01T17:36:48.133542+0000","last_became_active":"2023-12-01T17:05:13.499129+0000","last_became_peered":"2023-12-01T17:05:13.499129+0000","last_unstale":"2023-12-01T17:36:48.133542+0000","last_undegraded":"2023-12-01T17:36:48.133542+0000","last_fullsized":"2023-12-01T17:36:48.133542+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:48.132123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:48.132123+0000","last_clean_scrub_stamp":"2023-12-01T17:36:48.132123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:07:01.107754+0000","scrub_duration":0.083408903000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1059,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:52.941682+0000","last_change":"2023-12-01T17:36:52.942402+0000","last_active":"2023-12-01T17:36:52.941682+0000","last_peered":"2023-12-01T17:36:52.941682+0000","last_clean":"2023-12-01T17:36:52.941682+0000","last_became_active":"2023-12-01T17:05:37.740514+0000","last_became_peered":"2023-12-01T17:05:37.740514+0000","last_unstale":"2023-12-01T17:36:52.941682+0000","last_undegraded":"2023-12-01T17:36:52.941682+0000","last_fullsized":"2023-12-01T17:36:52.941682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:52.940307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:52.940307+0000","last_clean_scrub_stamp":"2023-12-01T17:36:52.940307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:02:53.085671+0000","scrub_duration":0.043691500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444427,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70172,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301668,"statfs":{"total":96636764160,"available":96564908032,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":20.553999999999998,"5min":12.125999999999999,"15min":15.801},"min":{"1min":6.5620000000000003,"5min":4.5899999999999999,"15min":4.5899999999999999},"max":{"1min":127.741,"5min":127.741,"15min":150.279},"last":9.7949999999999999},{"interface":"front","average":{"1min":29.547000000000001,"5min":14.099,"15min":17.922999999999998},"min":{"1min":8.5419999999999998,"5min":4.7229999999999999,"15min":4.7229999999999999},"max":{"1min":184.30600000000001,"5min":184.30600000000001,"15min":184.30600000000001},"last":10.425000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":18.321999999999999,"5min":12.057,"15min":16.654},"min":{"1min":6.5709999999999997,"5min":5.4610000000000003,"15min":5.4610000000000003},"max":{"1min":99.156000000000006,"5min":99.156000000000006,"15min":208.18199999999999},"last":8.0939999999999994},{"interface":"front","average":{"1min":34.377000000000002,"5min":16.986999999999998,"15min":19.702000000000002},"min":{"1min":7.9260000000000002,"5min":6.8769999999999998,"15min":6.2030000000000003},"max":{"1min":216.785,"5min":216.785,"15min":216.785},"last":12.215}]},{"osd":3,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":28.09,"5min":16.707999999999998,"15min":21.498000000000001},"min":{"1min":8.9960000000000004,"5min":7.8550000000000004,"15min":7.8550000000000004},"max":{"1min":243.88999999999999,"5min":243.88999999999999,"15min":276.07400000000001},"last":17.420999999999999},{"interface":"front","average":{"1min":30.187000000000001,"5min":19.353999999999999,"15min":24.324000000000002},"min":{"1min":13.336,"5min":10.973000000000001,"15min":7.2649999999999997},"max":{"1min":239.40000000000001,"5min":239.40000000000001,"15min":252.43299999999999},"last":12.879}]},{"osd":4,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":31.789000000000001,"5min":16.448,"15min":17.77},"min":{"1min":8.0429999999999993,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":230.511,"5min":230.511,"15min":230.511},"last":13.478999999999999},{"interface":"front","average":{"1min":33.975000000000001,"5min":17.349,"15min":21.777999999999999},"min":{"1min":7.6379999999999999,"5min":7.3979999999999997,"15min":7.3979999999999997},"max":{"1min":241.28100000000001,"5min":241.28100000000001,"15min":241.28100000000001},"last":11.558}]},{"osd":5,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":27.834,"5min":16.713999999999999,"15min":18.786000000000001},"min":{"1min":6.859,"5min":6.859,"15min":6.149},"max":{"1min":191.78700000000001,"5min":191.78700000000001,"15min":191.78700000000001},"last":18.361000000000001},{"interface":"front","average":{"1min":17.850999999999999,"5min":15.477,"15min":22.440999999999999},"min":{"1min":7.3769999999999998,"5min":7.3769999999999998,"15min":7.0270000000000001},"max":{"1min":111.208,"5min":111.208,"15min":311.13299999999998},"last":14.132999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":22.614999999999998,"5min":17.094999999999999,"15min":18.754999999999999},"min":{"1min":11.262,"5min":10.199,"15min":7.0960000000000001},"max":{"1min":94.787999999999997,"5min":94.787999999999997,"15min":99.623000000000005},"last":6.9299999999999997},{"interface":"front","average":{"1min":26.064,"5min":17.611000000000001,"15min":21.722999999999999},"min":{"1min":8.6600000000000001,"5min":8.6080000000000005,"15min":7.9909999999999997},"max":{"1min":153.83799999999999,"5min":153.83799999999999,"15min":186.11000000000001},"last":14.755000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":35.716000000000001,"5min":20.347000000000001,"15min":24.768999999999998},"min":{"1min":12.414999999999999,"5min":9.4819999999999993,"15min":9.1150000000000002},"max":{"1min":207.64599999999999,"5min":207.64599999999999,"15min":207.64599999999999},"last":19.218},{"interface":"front","average":{"1min":23.535,"5min":17.969999999999999,"15min":21.334},"min":{"1min":11.978999999999999,"5min":9.4469999999999992,"15min":9.0069999999999997},"max":{"1min":74.777000000000001,"5min":74.777000000000001,"15min":148.89599999999999},"last":15.449999999999999}]}]},{"osd":3,"up_from":16,"seq":68719477131,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69600,"kb_used_data":6488,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302240,"statfs":{"total":96636764160,"available":96565493760,"internally_reserved":0,"allocated":6643712,"data_stored":3599218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":38.427,"5min":15.26,"15min":18.015000000000001},"min":{"1min":6.8440000000000003,"5min":5.5540000000000003,"15min":5.2539999999999996},"max":{"1min":180.16,"5min":180.16,"15min":181.94200000000001},"last":7.056},{"interface":"front","average":{"1min":28.507000000000001,"5min":13.943,"15min":18.838000000000001},"min":{"1min":4.915,"5min":4.915,"15min":4.915},"max":{"1min":107.5,"5min":107.5,"15min":427.36500000000001},"last":11.792999999999999}]},{"osd":1,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":30.577999999999999,"5min":15.76,"15min":20.832999999999998},"min":{"1min":8.1419999999999995,"5min":7.585,"15min":6.4039999999999999},"max":{"1min":146.48099999999999,"5min":146.48099999999999,"15min":227.309},"last":8.3439999999999994},{"interface":"front","average":{"1min":33.158000000000001,"5min":15.33,"15min":19.213999999999999},"min":{"1min":8.2870000000000008,"5min":6.9850000000000003,"15min":5.827},"max":{"1min":223.179,"5min":223.179,"15min":421.40600000000001},"last":14.521000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":20.815999999999999,"5min":13.278,"15min":22.062000000000001},"min":{"1min":7.9829999999999997,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":86.969999999999999,"5min":86.969999999999999,"15min":451.97899999999998},"last":9.2940000000000005},{"interface":"front","average":{"1min":29.763999999999999,"5min":15.894,"15min":23.613},"min":{"1min":6.8540000000000001,"5min":6.7190000000000003,"15min":6.5170000000000003},"max":{"1min":223.84,"5min":223.84,"15min":384.58699999999999},"last":15.159000000000001}]},{"osd":4,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":33.475999999999999,"5min":17.899999999999999,"15min":21.863},"min":{"1min":9.6440000000000001,"5min":8.5640000000000001,"15min":6.5330000000000004},"max":{"1min":244.989,"5min":244.989,"15min":424.09500000000003},"last":9.9760000000000009},{"interface":"front","average":{"1min":31.292000000000002,"5min":17.065000000000001,"15min":21.056000000000001},"min":{"1min":8.9700000000000006,"5min":7.9870000000000001,"15min":7.0049999999999999},"max":{"1min":159.78200000000001,"5min":159.78200000000001,"15min":202.70500000000001},"last":15.484}]},{"osd":5,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":32.497,"5min":19.154,"15min":25.114000000000001},"min":{"1min":10.481999999999999,"5min":9.7400000000000002,"15min":9.0860000000000003},"max":{"1min":228.96700000000001,"5min":228.96700000000001,"15min":424.80099999999999},"last":17.558},{"interface":"front","average":{"1min":23.297999999999998,"5min":15.457000000000001,"15min":21.076000000000001},"min":{"1min":8.0359999999999996,"5min":7.2850000000000001,"15min":7.1710000000000003},"max":{"1min":87.635000000000005,"5min":87.635000000000005,"15min":231.65199999999999},"last":19.492999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":27.949000000000002,"5min":17.577000000000002,"15min":21.687999999999999},"min":{"1min":8.2859999999999996,"5min":8.2859999999999996,"15min":7.9340000000000002},"max":{"1min":136.684,"5min":136.684,"15min":203.99100000000001},"last":16.143000000000001},{"interface":"front","average":{"1min":30.741,"5min":19.934000000000001,"15min":25.875},"min":{"1min":12.468999999999999,"5min":10.978,"15min":10.978},"max":{"1min":147.88200000000001,"5min":147.88200000000001,"15min":203.34100000000001},"last":18.521000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":25.036000000000001,"5min":16.992999999999999,"15min":22.780000000000001},"min":{"1min":8.7509999999999994,"5min":7.9589999999999996,"15min":7.0259999999999998},"max":{"1min":105.342,"5min":105.342,"15min":196.05699999999999},"last":12.443},{"interface":"front","average":{"1min":27.623000000000001,"5min":19.024999999999999,"15min":25.555},"min":{"1min":11.186,"5min":11.186,"15min":10.492000000000001},"max":{"1min":102.29000000000001,"5min":102.29000000000001,"15min":236.84700000000001},"last":13.141999999999999}]}]},{"osd":2,"up_from":16,"seq":68719477129,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69604,"kb_used_data":6492,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302236,"statfs":{"total":96636764160,"available":96565489664,"internally_reserved":0,"allocated":6647808,"data_stored":3599223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.219000000000001,"5min":14.260999999999999,"15min":20.582999999999998},"min":{"1min":6.4989999999999997,"5min":4.7110000000000003,"15min":4.6059999999999999},"max":{"1min":169.971,"5min":169.971,"15min":231.76499999999999},"last":5.8620000000000001},{"interface":"front","average":{"1min":29.649999999999999,"5min":14.412000000000001,"15min":19.628},"min":{"1min":5.3049999999999997,"5min":5.1890000000000001,"15min":4.8810000000000002},"max":{"1min":150.74199999999999,"5min":150.74199999999999,"15min":222.62},"last":13.586}]},{"osd":1,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":23.131,"5min":12.741,"15min":20.030000000000001},"min":{"1min":6.4980000000000002,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":182.83600000000001,"5min":182.83600000000001,"15min":220.96700000000001},"last":9.1929999999999996},{"interface":"front","average":{"1min":27.039999999999999,"5min":16.943999999999999,"15min":23.343},"min":{"1min":8.6869999999999994,"5min":8.6869999999999994,"15min":7.8639999999999999},"max":{"1min":196.886,"5min":196.886,"15min":213.27799999999999},"last":11.282}]},{"osd":3,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":40.603999999999999,"5min":17.788,"15min":22.448},"min":{"1min":7.0609999999999999,"5min":7.0609999999999999,"15min":6.734},"max":{"1min":177.04900000000001,"5min":177.04900000000001,"15min":219.51300000000001},"last":9.8670000000000009},{"interface":"front","average":{"1min":41.942999999999998,"5min":19.474,"15min":24.347000000000001},"min":{"1min":7.6740000000000004,"5min":7.6740000000000004,"15min":6.9900000000000002},"max":{"1min":204.721,"5min":204.721,"15min":204.721},"last":15.445}]},{"osd":4,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":30.542000000000002,"5min":16.068000000000001,"15min":21.274000000000001},"min":{"1min":8.6310000000000002,"5min":7.2060000000000004,"15min":6.6799999999999997},"max":{"1min":180.369,"5min":180.369,"15min":251.745},"last":16.346},{"interface":"front","average":{"1min":33.243000000000002,"5min":18.972000000000001,"15min":24.882999999999999},"min":{"1min":10.202999999999999,"5min":10.202999999999999,"15min":7.5720000000000001},"max":{"1min":170.637,"5min":170.637,"15min":287.32100000000003},"last":18.129000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:36:09 2023","interfaces":[{"interface":"back","average":{"1min":37.953000000000003,"5min":18.436,"15min":24.466999999999999},"min":{"1min":8.5060000000000002,"5min":8.3300000000000001,"15min":8.0020000000000007},"max":{"1min":206.56700000000001,"5min":206.56700000000001,"15min":210.012},"last":12.079000000000001},{"interface":"front","average":{"1min":33.348999999999997,"5min":19.402000000000001,"15min":25.024000000000001},"min":{"1min":9.1980000000000004,"5min":8.1059999999999999,"15min":8.1059999999999999},"max":{"1min":232.91900000000001,"5min":232.91900000000001,"15min":232.91900000000001},"last":19.599}]},{"osd":6,"last update":"Fri Dec 1 17:36:05 2023","interfaces":[{"interface":"back","average":{"1min":35.573999999999998,"5min":19.463000000000001,"15min":23.882999999999999},"min":{"1min":8.8710000000000004,"5min":8.8710000000000004,"15min":8.1829999999999998},"max":{"1min":182.07400000000001,"5min":182.07400000000001,"15min":237.21899999999999},"last":10.603999999999999},{"interface":"front","average":{"1min":38.033999999999999,"5min":20.099,"15min":27.759},"min":{"1min":12.598000000000001,"5min":9.5960000000000001,"15min":9.5649999999999995},"max":{"1min":185.90299999999999,"5min":185.90299999999999,"15min":241.92699999999999},"last":12.826000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:50 2023","interfaces":[{"interface":"back","average":{"1min":34.993000000000002,"5min":20.754000000000001,"15min":24.891999999999999},"min":{"1min":8.9269999999999996,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":181.03,"5min":181.03,"15min":341.73000000000002},"last":16.975000000000001},{"interface":"front","average":{"1min":38.670000000000002,"5min":23.158999999999999,"15min":28.199999999999999},"min":{"1min":11.782,"5min":10.853,"15min":10.853},"max":{"1min":184.22999999999999,"5min":184.22999999999999,"15min":343.17399999999998},"last":18.789000000000001}]}]},{"osd":0,"up_from":15,"seq":64424509834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69972,"kb_used_data":6860,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301868,"statfs":{"total":96636764160,"available":96565112832,"internally_reserved":0,"allocated":7024640,"data_stored":3846006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 17:36:15 2023","interfaces":[{"interface":"back","average":{"1min":11.167,"5min":10.766999999999999,"15min":17.079000000000001},"min":{"1min":5.5579999999999998,"5min":5.5579999999999998,"15min":4.3550000000000004},"max":{"1min":33.322000000000003,"5min":33.322000000000003,"15min":210.43299999999999},"last":10.406000000000001},{"interface":"front","average":{"1min":12.661,"5min":10.788,"15min":18.704999999999998},"min":{"1min":6.242,"5min":6.0750000000000002,"15min":5.2590000000000003},"max":{"1min":44.020000000000003,"5min":44.020000000000003,"15min":257.30500000000001},"last":9.1430000000000007}]},{"osd":2,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.808,"5min":14.16,"15min":19.648},"min":{"1min":8.2010000000000005,"5min":7.4740000000000002,"15min":6.452},"max":{"1min":94.155000000000001,"5min":94.155000000000001,"15min":239.24600000000001},"last":16.588000000000001},{"interface":"front","average":{"1min":16.765000000000001,"5min":13.634,"15min":18.114999999999998},"min":{"1min":8.3179999999999996,"5min":6.8579999999999997,"15min":5.6520000000000001},"max":{"1min":86.099999999999994,"5min":86.099999999999994,"15min":246.28299999999999},"last":8.5350000000000001}]},{"osd":3,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":12.295999999999999,"15min":19.577000000000002},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"max":{"1min":128.08099999999999,"5min":128.08099999999999,"15min":240.017},"last":14.231},{"interface":"front","average":{"1min":28.396000000000001,"5min":17.071999999999999,"15min":25.015999999999998},"min":{"1min":9.452,"5min":8.3499999999999996,"15min":7.2779999999999996},"max":{"1min":136.40600000000001,"5min":136.40600000000001,"15min":249.86099999999999},"last":18.331}]},{"osd":4,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":15.007999999999999,"5min":12.295,"15min":18.792999999999999},"min":{"1min":7.5780000000000003,"5min":6.6909999999999998,"15min":6.6909999999999998},"max":{"1min":77.429000000000002,"5min":77.429000000000002,"15min":241.292},"last":7.6020000000000003},{"interface":"front","average":{"1min":19.893999999999998,"5min":16.827000000000002,"15min":22.696000000000002},"min":{"1min":11.875,"5min":9.3759999999999994,"15min":7.2290000000000001},"max":{"1min":91.792000000000002,"5min":91.792000000000002,"15min":236.65799999999999},"last":19.152000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":16.934000000000001,"5min":14.622,"15min":24.321999999999999},"min":{"1min":9.2949999999999999,"5min":8.6929999999999996,"15min":7.9400000000000004},"max":{"1min":45.228000000000002,"5min":45.228000000000002,"15min":241.91200000000001},"last":15.053000000000001},{"interface":"front","average":{"1min":16.408000000000001,"5min":14.084,"15min":21.462},"min":{"1min":7.2779999999999996,"5min":7.2779999999999996,"15min":6.4279999999999999},"max":{"1min":66.477000000000004,"5min":66.477000000000004,"15min":247.185},"last":12.249000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":17.015000000000001,"5min":15.752000000000001,"15min":23.873999999999999},"min":{"1min":9.8810000000000002,"5min":8.2829999999999995,"15min":7.923},"max":{"1min":50.194000000000003,"5min":50.194000000000003,"15min":244.93799999999999},"last":15.791},{"interface":"front","average":{"1min":16.581,"5min":15.106999999999999,"15min":22.677},"min":{"1min":9.2609999999999992,"5min":9.2110000000000003,"15min":7.734},"max":{"1min":42.387999999999998,"5min":42.387999999999998,"15min":247.80500000000001},"last":11.055999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:36:08 2023","interfaces":[{"interface":"back","average":{"1min":18.469999999999999,"5min":17.091999999999999,"15min":24.052},"min":{"1min":11.074999999999999,"5min":10.281000000000001,"15min":9.0820000000000007},"max":{"1min":37.758000000000003,"5min":37.758000000000003,"15min":238.58099999999999},"last":17.507999999999999},{"interface":"front","average":{"1min":18.225000000000001,"5min":16.898,"15min":24.481000000000002},"min":{"1min":11.706,"5min":9.3140000000000001,"15min":9.3140000000000001},"max":{"1min":36.850000000000001,"5min":36.850000000000001,"15min":273.142},"last":11.659000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542540,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69604,"kb_used_data":6492,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302236,"statfs":{"total":96636764160,"available":96565489664,"internally_reserved":0,"allocated":6647808,"data_stored":3599223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":10.34,"5min":9.7390000000000008,"15min":13.387},"min":{"1min":4.8339999999999996,"5min":4.577,"15min":4.0730000000000004},"max":{"1min":25.731999999999999,"5min":25.731999999999999,"15min":129.89699999999999},"last":10.462},{"interface":"front","average":{"1min":11.954000000000001,"5min":10.832000000000001,"15min":15.518000000000001},"min":{"1min":6.5010000000000003,"5min":6.4669999999999996,"15min":4.8129999999999997},"max":{"1min":34.526000000000003,"5min":34.526000000000003,"15min":155.99100000000001},"last":7.0620000000000003}]},{"osd":1,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":14.457000000000001,"5min":12.138,"15min":15.375},"min":{"1min":7.9969999999999999,"5min":6.702,"15min":6.2789999999999999},"max":{"1min":50.130000000000003,"5min":50.130000000000003,"15min":175.55000000000001},"last":9.7520000000000007},{"interface":"front","average":{"1min":18.449000000000002,"5min":15.638999999999999,"15min":19.158000000000001},"min":{"1min":10.378,"5min":10.378,"15min":7.0490000000000004},"max":{"1min":54.060000000000002,"5min":54.060000000000002,"15min":130.91999999999999},"last":12.499000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":21.776,"5min":15.327,"15min":19.893999999999998},"min":{"1min":7.0010000000000003,"5min":6.8879999999999999,"15min":6.8879999999999999},"max":{"1min":172.459,"5min":172.459,"15min":191.71299999999999},"last":8.9480000000000004},{"interface":"front","average":{"1min":19.948,"5min":11.276999999999999,"15min":16.713000000000001},"min":{"1min":4.4370000000000003,"5min":4.4370000000000003,"15min":4.4370000000000003},"max":{"1min":174.608,"5min":174.608,"15min":244.46299999999999},"last":16.611000000000001}]},{"osd":3,"last update":"Fri Dec 1 17:36:01 2023","interfaces":[{"interface":"back","average":{"1min":19.504999999999999,"5min":15.193,"15min":20.762},"min":{"1min":10.653,"5min":8.2929999999999993,"15min":6.5830000000000002},"max":{"1min":70.066999999999993,"5min":70.066999999999993,"15min":223.97800000000001},"last":11.766999999999999},{"interface":"front","average":{"1min":18.291,"5min":14.847,"15min":22.268000000000001},"min":{"1min":7.2080000000000002,"5min":7.2080000000000002,"15min":6.8559999999999999},"max":{"1min":71.659999999999997,"5min":71.659999999999997,"15min":212.309},"last":15.843}]},{"osd":4,"last update":"Fri Dec 1 17:36:51 2023","interfaces":[{"interface":"back","average":{"1min":22.206,"5min":16.388000000000002,"15min":19.161000000000001},"min":{"1min":9.109,"5min":8.9860000000000007,"15min":7.0510000000000002},"max":{"1min":67.287000000000006,"5min":67.287000000000006,"15min":134.95699999999999},"last":13.775},{"interface":"front","average":{"1min":25.456,"5min":19.428999999999998,"15min":22.629999999999999},"min":{"1min":13.336,"5min":11.27,"15min":9.9809999999999999},"max":{"1min":74.200999999999993,"5min":74.200999999999993,"15min":135.798},"last":17.378}]},{"osd":5,"last update":"Fri Dec 1 17:36:51 2023","interfaces":[{"interface":"back","average":{"1min":21.643000000000001,"5min":16.739000000000001,"15min":21.106000000000002},"min":{"1min":7.9909999999999997,"5min":7.8040000000000003,"15min":7.2869999999999999},"max":{"1min":70.823999999999998,"5min":70.823999999999998,"15min":288.577},"last":11.157},{"interface":"front","average":{"1min":26.931000000000001,"5min":19.710000000000001,"15min":23.295999999999999},"min":{"1min":12.140000000000001,"5min":12.140000000000001,"15min":10.234},"max":{"1min":96.646000000000001,"5min":96.646000000000001,"15min":142.77699999999999},"last":18.596}]},{"osd":7,"last update":"Fri Dec 1 17:36:51 2023","interfaces":[{"interface":"back","average":{"1min":16.306999999999999,"5min":11.847,"15min":15.145},"min":{"1min":6.5309999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"max":{"1min":59.734000000000002,"5min":59.734000000000002,"15min":168.59700000000001},"last":8.3179999999999996},{"interface":"front","average":{"1min":23.439,"5min":18.123000000000001,"15min":21.085000000000001},"min":{"1min":14.02,"5min":8.9459999999999997,"15min":8.4770000000000003},"max":{"1min":69.829999999999998,"5min":69.829999999999998,"15min":183.054},"last":13.109}]}]},{"osd":5,"up_from":14,"seq":60129542541,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74332,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297508,"statfs":{"total":96636764160,"available":96560648192,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":24.530999999999999,"5min":11.712,"15min":15.721},"min":{"1min":4.3620000000000001,"5min":4.1360000000000001,"15min":4.1360000000000001},"max":{"1min":201.333,"5min":201.333,"15min":228.261},"last":6.8620000000000001},{"interface":"front","average":{"1min":27.792000000000002,"5min":13.209,"15min":17.690999999999999},"min":{"1min":6.5709999999999997,"5min":5.0339999999999998,"15min":4.6580000000000004},"max":{"1min":221.876,"5min":221.876,"15min":222.441},"last":9.7949999999999999}]},{"osd":1,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.556000000000001,"5min":12.199999999999999,"15min":19.036000000000001},"min":{"1min":7.0490000000000004,"5min":6.0640000000000001,"15min":4.8899999999999997},"max":{"1min":66.040999999999997,"5min":66.040999999999997,"15min":216.20099999999999},"last":8.0939999999999994},{"interface":"front","average":{"1min":17.574000000000002,"5min":12.794,"15min":19.777000000000001},"min":{"1min":7.7430000000000003,"5min":6.5650000000000004,"15min":5.6210000000000004},"max":{"1min":65.156999999999996,"5min":65.156999999999996,"15min":255.95599999999999},"last":14.291}]},{"osd":2,"last update":"Fri Dec 1 17:35:58 2023","interfaces":[{"interface":"back","average":{"1min":12.641999999999999,"5min":11.792,"15min":17.571999999999999},"min":{"1min":6.468,"5min":6.468,"15min":6.468},"max":{"1min":39.947000000000003,"5min":39.947000000000003,"15min":179.81999999999999},"last":10.816000000000001},{"interface":"front","average":{"1min":16.623999999999999,"5min":15.505000000000001,"15min":22.356999999999999},"min":{"1min":8.7070000000000007,"5min":8.7070000000000007,"15min":8.7070000000000007},"max":{"1min":42.759,"5min":42.759,"15min":224.47300000000001},"last":13.596}]},{"osd":3,"last update":"Fri Dec 1 17:36:03 2023","interfaces":[{"interface":"back","average":{"1min":17.655000000000001,"5min":13.112,"15min":19.452999999999999},"min":{"1min":6.819,"5min":6.819,"15min":5.9020000000000001},"max":{"1min":61.933,"5min":61.933,"15min":248.07300000000001},"last":12.992000000000001},{"interface":"front","average":{"1min":22.122,"5min":16.693000000000001,"15min":23.396000000000001},"min":{"1min":11.053000000000001,"5min":10.221,"15min":9.7989999999999995},"max":{"1min":62.886000000000003,"5min":62.886000000000003,"15min":235.02699999999999},"last":18.088000000000001}]},{"osd":4,"last update":"Fri Dec 1 17:36:55 2023","interfaces":[{"interface":"back","average":{"1min":19.247,"5min":16.109000000000002,"15min":19.242000000000001},"min":{"1min":8.3200000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"max":{"1min":36.350999999999999,"5min":59.747,"15min":211.09399999999999},"last":15.548999999999999},{"interface":"front","average":{"1min":22.109999999999999,"5min":15.996,"15min":18.448},"min":{"1min":9.2189999999999994,"5min":8.6850000000000005,"15min":7.0540000000000003},"max":{"1min":114.05200000000001,"5min":114.05200000000001,"15min":216.85300000000001},"last":9.2189999999999994}]},{"osd":6,"last update":"Fri Dec 1 17:36:55 2023","interfaces":[{"interface":"back","average":{"1min":22.838999999999999,"5min":18.434999999999999,"15min":21.120000000000001},"min":{"1min":10.314,"5min":10.103,"15min":8.4339999999999993},"max":{"1min":81.772999999999996,"5min":111.396,"15min":223.89500000000001},"last":17.256},{"interface":"front","average":{"1min":19.198,"5min":16.227,"15min":14.467000000000001},"min":{"1min":8.0660000000000007,"5min":7.0220000000000002,"15min":5.6479999999999997},"max":{"1min":59.250999999999998,"5min":59.250999999999998,"15min":159.84399999999999},"last":11.396000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:55 2023","interfaces":[{"interface":"back","average":{"1min":23.693999999999999,"5min":18.780999999999999,"15min":21.957000000000001},"min":{"1min":10.323,"5min":10.323,"15min":9.6430000000000007},"max":{"1min":93.954999999999998,"5min":93.954999999999998,"15min":226.06100000000001},"last":12.390000000000001},{"interface":"front","average":{"1min":21.358000000000001,"5min":18.934000000000001,"15min":21.984999999999999},"min":{"1min":8.9890000000000008,"5min":8.9890000000000008,"15min":8.9559999999999995},"max":{"1min":49.155000000000001,"5min":125.431,"15min":317.267},"last":16.420999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542541,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74132,"kb_used_data":6860,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297708,"statfs":{"total":96636764160,"available":96560852992,"internally_reserved":0,"allocated":7024640,"data_stored":3846006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":7.8300000000000001,"5min":9.3949999999999996,"15min":16.763000000000002},"min":{"1min":4.1109999999999998,"5min":4.1109999999999998,"15min":3.7690000000000001},"max":{"1min":15.778,"5min":24.963999999999999,"15min":521.55899999999997},"last":7.1059999999999999},{"interface":"front","average":{"1min":9.3529999999999998,"5min":9.8160000000000007,"15min":15.084},"min":{"1min":6.3360000000000003,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":13.864000000000001,"5min":21.491,"15min":281.125},"last":9.1359999999999992}]},{"osd":1,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":11.353999999999999,"5min":11.315,"15min":19.023},"min":{"1min":8.2140000000000004,"5min":6.7969999999999997,"15min":6.2770000000000001},"max":{"1min":14.935,"5min":21.539000000000001,"15min":308.99299999999999},"last":8.2070000000000007},{"interface":"front","average":{"1min":16.753,"5min":15.666,"15min":22.716000000000001},"min":{"1min":10.832000000000001,"5min":9.1069999999999993,"15min":6.3079999999999998},"max":{"1min":31.140000000000001,"5min":31.140000000000001,"15min":289.79399999999998},"last":12.34}]},{"osd":2,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":16.849,"5min":15.061999999999999,"15min":21.503},"min":{"1min":11.24,"5min":9.1029999999999998,"15min":5.976},"max":{"1min":27.797999999999998,"5min":27.797999999999998,"15min":298.18200000000002},"last":10.519},{"interface":"front","average":{"1min":12.192,"5min":10.987,"15min":16.457999999999998},"min":{"1min":8.0640000000000001,"5min":6.4820000000000002,"15min":6.4820000000000002},"max":{"1min":25.771999999999998,"5min":25.771999999999998,"15min":223.768},"last":13.006}]},{"osd":3,"last update":"Fri Dec 1 17:36:10 2023","interfaces":[{"interface":"back","average":{"1min":12.936999999999999,"5min":12.680999999999999,"15min":18.734000000000002},"min":{"1min":8.6850000000000005,"5min":7.1580000000000004,"15min":6.2169999999999996},"max":{"1min":26.462,"5min":26.462,"15min":243.40299999999999},"last":9.8970000000000002},{"interface":"front","average":{"1min":17.468,"5min":16.356999999999999,"15min":22.004000000000001},"min":{"1min":13.864000000000001,"5min":10.156000000000001,"15min":8.4390000000000001},"max":{"1min":37.262999999999998,"5min":37.262999999999998,"15min":253.56100000000001},"last":14.307}]},{"osd":5,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":17.428999999999998,"5min":15.695,"15min":22.879999999999999},"min":{"1min":10.804,"5min":9.5199999999999996,"15min":6.6349999999999998},"max":{"1min":35.213999999999999,"5min":35.213999999999999,"15min":336.55599999999998},"last":13.593},{"interface":"front","average":{"1min":10.366,"5min":11.09,"15min":19.364000000000001},"min":{"1min":7.0019999999999998,"5min":6.4859999999999998,"15min":6.4790000000000001},"max":{"1min":14.853,"5min":23.786000000000001,"15min":357.50900000000001},"last":11.151999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":14.206,"5min":15.972,"15min":20.510999999999999},"min":{"1min":9.2010000000000005,"5min":9.2010000000000005,"15min":8.1080000000000005},"max":{"1min":34.234000000000002,"5min":34.234000000000002,"15min":255.52500000000001},"last":11.736000000000001},{"interface":"front","average":{"1min":19.222000000000001,"5min":15.115,"15min":19.218},"min":{"1min":15.247999999999999,"5min":7.7080000000000002,"15min":7.7080000000000002},"max":{"1min":37.884,"5min":37.884,"15min":312.791},"last":14.906000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:36:04 2023","interfaces":[{"interface":"back","average":{"1min":15.161,"5min":15.696,"15min":23.895},"min":{"1min":9.7590000000000003,"5min":9.6940000000000008,"15min":8.7330000000000005},"max":{"1min":23.195,"5min":23.195,"15min":413.48899999999998},"last":18.021000000000001},{"interface":"front","average":{"1min":17.57,"5min":16.561,"15min":21.308},"min":{"1min":9.4770000000000003,"5min":9.0809999999999995,"15min":6.6820000000000004},"max":{"1min":30.414999999999999,"5min":30.414999999999999,"15min":417.952},"last":15.617000000000001}]}]},{"osd":7,"up_from":13,"seq":55834575247,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69780,"kb_used_data":6668,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302060,"statfs":{"total":96636764160,"available":96565309440,"internally_reserved":0,"allocated":6828032,"data_stored":3717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":13.364000000000001,"5min":11.880000000000001,"15min":16.117000000000001},"min":{"1min":8.8580000000000005,"5min":5.2729999999999997,"15min":4.3360000000000003},"max":{"1min":39.423000000000002,"5min":39.423000000000002,"15min":203.19300000000001},"last":14.029},{"interface":"front","average":{"1min":17.373999999999999,"5min":15.071999999999999,"15min":18.241},"min":{"1min":11.513,"5min":6.4269999999999996,"15min":4.4199999999999999},"max":{"1min":40.698999999999998,"5min":40.698999999999998,"15min":204.13},"last":15.602}]},{"osd":1,"last update":"Fri Dec 1 17:35:57 2023","interfaces":[{"interface":"back","average":{"1min":7.9080000000000004,"5min":8.2289999999999992,"15min":15.226000000000001},"min":{"1min":6.1139999999999999,"5min":5.2300000000000004,"15min":5.0369999999999999},"max":{"1min":10.85,"5min":17.564,"15min":167.82900000000001},"last":9.4629999999999992},{"interface":"front","average":{"1min":15.026,"5min":14.375999999999999,"15min":16.369},"min":{"1min":11.634,"5min":7.4870000000000001,"15min":5.484},"max":{"1min":22.280999999999999,"5min":22.280999999999999,"15min":154.137},"last":17.16}]},{"osd":2,"last update":"Fri Dec 1 17:36:23 2023","interfaces":[{"interface":"back","average":{"1min":11.753,"5min":10.013999999999999,"15min":16.608000000000001},"min":{"1min":5.8700000000000001,"5min":5.1820000000000004,"15min":5.1820000000000004},"max":{"1min":26.274999999999999,"5min":26.274999999999999,"15min":230.40600000000001},"last":6.8369999999999997},{"interface":"front","average":{"1min":14.275,"5min":12.394,"15min":17.012},"min":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"max":{"1min":29.861000000000001,"5min":29.861000000000001,"15min":268.68000000000001},"last":11.474}]},{"osd":3,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":13.083,"15min":20.297000000000001},"min":{"1min":8.1359999999999992,"5min":6.5110000000000001,"15min":6.5110000000000001},"max":{"1min":85.724000000000004,"5min":85.724000000000004,"15min":290.15300000000002},"last":10.618},{"interface":"front","average":{"1min":14.601000000000001,"5min":11.683,"15min":20.149000000000001},"min":{"1min":6.6319999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"max":{"1min":83.183000000000007,"5min":83.183000000000007,"15min":305.41399999999999},"last":8.7149999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:40 2023","interfaces":[{"interface":"back","average":{"1min":22.085999999999999,"5min":15.728999999999999,"15min":18.579000000000001},"min":{"1min":10.250999999999999,"5min":8.5489999999999995,"15min":6.9299999999999997},"max":{"1min":108.761,"5min":108.761,"15min":149.93000000000001},"last":14.84},{"interface":"front","average":{"1min":25.981999999999999,"5min":19.062999999999999,"15min":23.542000000000002},"min":{"1min":13.847,"5min":11.496,"15min":9.7810000000000006},"max":{"1min":109.95399999999999,"5min":109.95399999999999,"15min":239.37},"last":18.471}]},{"osd":5,"last update":"Fri Dec 1 17:36:40 2023","interfaces":[{"interface":"back","average":{"1min":30.864000000000001,"5min":17.353999999999999,"15min":21.384},"min":{"1min":11.603999999999999,"5min":7.3879999999999999,"15min":6.8460000000000001},"max":{"1min":124.021,"5min":124.021,"15min":282.31200000000001},"last":16.404},{"interface":"front","average":{"1min":29.943999999999999,"5min":16.628,"15min":20.617999999999999},"min":{"1min":7.1900000000000004,"5min":6.1529999999999996,"15min":6.1529999999999996},"max":{"1min":133.404,"5min":133.404,"15min":193.83600000000001},"last":12.231999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:40 2023","interfaces":[{"interface":"back","average":{"1min":26.922000000000001,"5min":19.552,"15min":20.542000000000002},"min":{"1min":13.866,"5min":11.816000000000001,"15min":6.625},"max":{"1min":90.069000000000003,"5min":90.069000000000003,"15min":209.25},"last":17.920999999999999},{"interface":"front","average":{"1min":35.006,"5min":22.48,"15min":25.449999999999999},"min":{"1min":16.302,"5min":14.266999999999999,"15min":8.9420000000000002},"max":{"1min":150.14400000000001,"5min":150.14400000000001,"15min":193.184},"last":19.536000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:36:59.138 INFO:tasks.ceph:pgid 22.6 last_scrub_stamp 2023-12-01T17:05:03.392191+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=5, tm_sec=3, tm_wday=4, tm_yday=335, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=1, tm_hour=17, tm_min=36, tm_sec=30, tm_wday=4, tm_yday=335, tm_isdst=0) 2023-12-01T17:36:59.139 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-01T17:37:00.169 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:13.951 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.170 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:13.952 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.180 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:13.962 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.214 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:13.974 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.215 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:13.976 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.224 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:13.985 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.237 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:14.018 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.239 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:14.021 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:00.250 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:14.032 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.206 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:28.987 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.208 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:28.990 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.220 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:29.002 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.254 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:29.014 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.257 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:29.039 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.258 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:29.018 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.258 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:29.040 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.272 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:29.033 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:15.274 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:29.056 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:19.140 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.250+0000 7f15d9aa2640 10 monclient: get_monmap_and_config 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.250+0000 7f15d9aa2640 10 monclient: build_initial_monmap 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.250+0000 7f15d9aa2640 10 monclient: monmap: 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:37:19.251397+0000 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:37:19.251397+0000 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:37:19.250 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:37:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:37:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:37:19.251 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:37:19.251 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.251+0000 7f15d9aa2640 1 Processor -- start 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.251+0000 7f15d9aa2640 1 -- start start 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.251+0000 7f15d9aa2640 10 monclient: init 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient: _reopen_session rank -1 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f15d40a8a20 0x7f15d413eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f15d40a8a20 addr v2:172.21.15.92:3300/0 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f15d40a8e00 addr v1:172.21.15.67:6789/0 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15d41423b0 addr v1:172.21.15.92:6789/0 2023-12-01T17:37:19.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40b21f0 con 0x7f15d40a8e00 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40a9ed0 con 0x7f15d41423b0 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d40a9cc0 con 0x7f15d40a8a20 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient(hunting): _renew_subs 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d9aa2640 10 monclient(hunting): authenticate will time out at 3370.489524s 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d2ffd640 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f15d40a8e00 0x7f15d413f3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:41054/0 (socket says 172.21.15.67:41054) 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.252+0000 7f15d2ffd640 1 -- 172.21.15.67:0/4135478862 learned_addr learned my addr 172.21.15.67:0/4135478862 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-12-01T17:37:19.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.253+0000 7f15d37fe640 1 --2- 172.21.15.67:0/4135478862 >> v2:172.21.15.92:3300/0 conn(0x7f15d40a8a20 0x7f15d413eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.253+0000 7f15d37fe640 10 monclient(hunting): get_auth_request con 0x7f15d40a8a20 auth_method 0 2023-12-01T17:37:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.253+0000 7f15d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.253+0000 7f15d37fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.253+0000 7f15d37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.254+0000 7f15d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.254+0000 7f15d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.254+0000 7f15d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.255+0000 7f15d27fc640 1 -- 172.21.15.67:0/4135478862 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671296799 0 0) 0x7f15d40b21f0 con 0x7f15d40a8e00 2023-12-01T17:37:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.255+0000 7f15d27fc640 10 monclient(hunting): _negotiate 2023-12-01T17:37:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.255+0000 7f15d27fc640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.255+0000 7f15d27fc640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.255+0000 7f15d27fc640 10 monclient(hunting): my global_id is 16074 2023-12-01T17:37:19.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.255+0000 7f15d27fc640 1 -- 172.21.15.67:0/4135478862 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c0003520 con 0x7f15d40a8e00 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 10 monclient(hunting): handle_auth_done global_id 23257 payload 274 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 1 -- 172.21.15.67:0/4135478862 >> v1:172.21.15.92:6789/0 conn(0x7f15d41423b0 legacy=0x7f15d4142840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 1 -- 172.21.15.67:0/4135478862 >> v1:172.21.15.67:6789/0 conn(0x7f15d40a8e00 legacy=0x7f15d413f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 10 monclient: _finish_hunting 0 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 1 monclient: found mon.noname-c 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.257+0000 7f15d37fe640 1 -- 172.21.15.67:0/4135478862 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d40aac70 con 0x7f15d40a8a20 2023-12-01T17:37:19.259 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.260+0000 7f15d37fe640 1 --2- 172.21.15.67:0/4135478862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d413eeb0 secure :-1 s=READY pgs=5918 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0097a0 tx=0x7f15bc02b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f2b0172842ca5de server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 1 -- 172.21.15.67:0/4135478862 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc02e070 con 0x7f15d40a8a20 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: dump: 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.264 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: _finish_auth 0 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: _check_auth_tickets 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 1 -- 172.21.15.67:0/4135478862 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc0315a0 con 0x7f15d40a8a20 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 1 -- 172.21.15.67:0/4135478862 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc031930 con 0x7f15d40a8a20 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d27fc640 10 monclient: dump: 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.265 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d9aa2640 5 monclient: authenticate success, global_id 23257 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d9aa2640 10 monclient: get_monmap_and_config success 2023-12-01T17:37:19.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.264+0000 7f15d9aa2640 4 set_mon_vals no callback set 2023-12-01T17:37:19.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 10 monclient: shutdown 2023-12-01T17:37:19.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:37:19.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 1 -- 172.21.15.67:0/4135478862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 msgr2=0x7f15d413eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/4135478862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d413eeb0 secure :-1 s=READY pgs=5918 cs=0 l=1 rev1=1 crypto rx=0x7f15bc0097a0 tx=0x7f15bc02b840 comp rx=0 tx=0).stop 2023-12-01T17:37:19.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 1 -- 172.21.15.67:0/4135478862 shutdown_connections 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/4135478862 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d413eeb0 unknown :-1 s=CLOSED pgs=5918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 1 -- 172.21.15.67:0/4135478862 >> 172.21.15.67:0/4135478862 conn(0x7f15d40b7870 msgr2=0x7f15d40b7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 1 -- 172.21.15.67:0/4135478862 shutdown_connections 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.265+0000 7f15d9aa2640 1 -- 172.21.15.67:0/4135478862 wait complete. 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 10 monclient: build_initial_monmap 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 10 monclient: monmap: 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:37:19.270 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 1 librados: starting msgr at 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 1 librados: starting objecter 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 1 Processor -- start 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 1 -- start start 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 1 librados: setting wanted keys 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 1 librados: calling monclient init 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.266+0000 7f15d9aa2640 10 monclient: init 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient: _reopen_session rank -1 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:37:19.271 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d4145cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): picked mon.noname-b con 0x7f15d40a8a20 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d40a8e00 0x7f15d40abc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): picked mon.noname-a con 0x7f15d40a8e00 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15d41423b0 0x7f15d413a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): picked mon.noname-c con 0x7f15d41423b0 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d413a380 con 0x7f15d40a8e00 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d413b5c0 con 0x7f15d40a8a20 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40ab510 con 0x7f15d41423b0 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): _renew_subs 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d9aa2640 10 monclient(hunting): authenticate will time out at 3370.504654s 2023-12-01T17:37:19.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d37fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d4145cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d2ffd640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d40a8e00 0x7f15d40abc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.267+0000 7f15d3fff640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15d41423b0 0x7f15d413a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d4145cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:49082/0 (socket says 172.21.15.67:49082) 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 1 -- 172.21.15.67:0/1900513341 learned_addr learned my addr 172.21.15.67:0/1900513341 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 10 monclient(hunting): get_auth_request con 0x7f15d40a8a20 auth_method 0 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d2ffd640 10 monclient(hunting): get_auth_request con 0x7f15d40a8e00 auth_method 0 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d2ffd640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d2ffd640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d3fff640 10 monclient(hunting): get_auth_request con 0x7f15d41423b0 auth_method 0 2023-12-01T17:37:19.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d3fff640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d3fff640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.268+0000 7f15d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 10 monclient(hunting): handle_auth_done global_id 23263 payload 824 2023-12-01T17:37:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 1 -- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15d41423b0 msgr2=0x7f15d413a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15d41423b0 0x7f15d413a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 1 -- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d40a8e00 msgr2=0x7f15d40abc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d40a8e00 0x7f15d40abc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 10 monclient: _finish_hunting 0 2023-12-01T17:37:19.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 1 monclient: found mon.noname-b 2023-12-01T17:37:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:37:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.271+0000 7f15d37fe640 1 -- 172.21.15.67:0/1900513341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15bc009400 con 0x7f15d40a8a20 2023-12-01T17:37:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.273+0000 7f15d37fe640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d4145cf0 secure :-1 s=READY pgs=5919 cs=0 l=1 rev1=1 crypto rx=0x7f15bc034040 tx=0x7f15bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:37:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.273+0000 7f15cbfff640 10 client.?.objecter ms_handle_connect 0x7f15d40a8a20 2023-12-01T17:37:19.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.273+0000 7f15cbfff640 10 client.?.objecter resend_mon_ops 2023-12-01T17:37:19.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 1 -- 172.21.15.67:0/1900513341 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc032840 con 0x7f15d40a8a20 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: dump: 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.278 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: _finish_auth 0 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: _check_auth_tickets 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 1 -- 172.21.15.67:0/1900513341 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15bc0329c0 con 0x7f15d40a8a20 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 5 monclient: authenticate success, global_id 23263 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 10 monclient: _renew_subs 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d40ab7f0 con 0x7f15d40a8a20 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 1 -- 172.21.15.67:0/1900513341 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15bc032d70 con 0x7f15d40a8a20 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15cbfff640 10 monclient: dump: 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.279 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d1ffb640 4 set_mon_vals no callback set 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 10 client.23263.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 10 monclient: _renew_subs 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d413ace0 con 0x7f15d40a8a20 2023-12-01T17:37:19.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.278+0000 7f15d9aa2640 1 librados: init done 2023-12-01T17:37:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.279+0000 7f15d9aa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:37:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.279+0000 7f15d9aa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:37:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.279+0000 7f15d9aa2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.279+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d413ae70 con 0x7f15d40a8a20 2023-12-01T17:37:19.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.297+0000 7f15cbfff640 1 -- 172.21.15.67:0/1900513341 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f15bc04e430 con 0x7f15d40a8a20 2023-12-01T17:37:19.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.298+0000 7f15cbfff640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f15b406c820 0x7f15b406ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.298+0000 7f15d2ffd640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f15b406c820 0x7f15b406ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.298+0000 7f15d2ffd640 10 monclient: get_auth_request con 0x7f15b406c820 auth_method 0 2023-12-01T17:37:19.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.298+0000 7f15cbfff640 10 client.23263.objecter ms_handle_connect 0x7f15b406c820 2023-12-01T17:37:19.298 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.298+0000 7f15d2ffd640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f15b406c820 0x7f15b406ece0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f15d40b61e0 tx=0x7f15c4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:37:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.299+0000 7f15cbfff640 1 -- 172.21.15.67:0/1900513341 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f15bc07a930 con 0x7f15d40a8a20 2023-12-01T17:37:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.299+0000 7f15cbfff640 10 client.23263.objecter ms_dispatch 0x7f15d4000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:37:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.299+0000 7f15cbfff640 3 client.23263.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:37:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.299+0000 7f15cbfff640 3 client.23263.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:37:19.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.300+0000 7f15cbfff640 20 client.23263.objecter dump_active .. 0 homeless 2023-12-01T17:37:19.532 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.533+0000 7f15cbfff640 1 -- 172.21.15.67:0/1900513341 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15bc076c70 con 0x7f15d40a8a20 2023-12-01T17:37:19.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.533+0000 7f15cbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:37:19.533 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.533+0000 7f15cbfff640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:37:19.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.711+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 --> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15d413da50 con 0x7f15b406c820 2023-12-01T17:37:19.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.716+0000 7f15cbfff640 1 -- 172.21.15.67:0/1900513341 <== mgr.4100 v2:172.21.15.67:6800/1043126838 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152245 (secure 0 0 0) 0x7f15d413da50 con 0x7f15b406c820 2023-12-01T17:37:19.717 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-12-01T17:37:19.718 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.718+0000 7f15d9aa2640 10 librados: watch_flush enter 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.718+0000 7f15d9aa2640 10 librados: watch_flush exit 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.718+0000 7f15d9aa2640 20 client.23263.objecter shutdown clearing up homeless session... 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.718+0000 7f15d9aa2640 10 client.23263.objecter successfully canceled tick 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f15b406c820 msgr2=0x7f15b406ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f15b406c820 0x7f15b406ece0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f15d40b61e0 tx=0x7f15c4009040 comp rx=0 tx=0).stop 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 10 monclient: shutdown 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 msgr2=0x7f15d4145cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d4145cf0 secure :-1 s=READY pgs=5919 cs=0 l=1 rev1=1 crypto rx=0x7f15bc034040 tx=0x7f15bc0046c0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 shutdown_connections 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f15b406c820 0x7f15b406ece0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.720 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f15d41423b0 0x7f15d413a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f15d40a8e00 0x7f15d40abc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 --2- 172.21.15.67:0/1900513341 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f15d40a8a20 0x7f15d4145cf0 unknown :-1 s=CLOSED pgs=5919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 >> 172.21.15.67:0/1900513341 conn(0x7f15d40b7870 msgr2=0x7f15d40b6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:37:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 shutdown_connections 2023-12-01T17:37:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 -- 172.21.15.67:0/1900513341 wait complete. 2023-12-01T17:37:19.721 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.719+0000 7f15d9aa2640 1 librados: shutdown 2023-12-01T17:37:19.729 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":2106,"stamp":"2023-12-01T17:37:18.785166+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":567196,"kb_used_data":53980,"kb_used_omap":12,"kb_used_meta":513139,"kb_avail":754407524,"statfs":{"total":773094113280,"available":772513304576,"internally_reserved":0,"allocated":55275520,"data_stored":30438252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":525454896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002841"},"pg_stats":[{"pgid":"22.6","version":"0'0","reported_seq":1058,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:54.984140+0000","last_change":"2023-12-01T17:36:54.984861+0000","last_active":"2023-12-01T17:36:54.984140+0000","last_peered":"2023-12-01T17:36:54.984140+0000","last_clean":"2023-12-01T17:36:54.984140+0000","last_became_active":"2023-12-01T17:05:14.257152+0000","last_became_peered":"2023-12-01T17:05:14.257152+0000","last_unstale":"2023-12-01T17:36:54.984140+0000","last_undegraded":"2023-12-01T17:36:54.984140+0000","last_fullsized":"2023-12-01T17:36:54.984140+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:54.982716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:54.982716+0000","last_clean_scrub_stamp":"2023-12-01T17:36:54.982716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:24:56.159914+0000","scrub_duration":0.039714343999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"0'0","reported_seq":1066,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:53.343956+0000","last_change":"2023-12-01T17:36:53.344682+0000","last_active":"2023-12-01T17:36:53.343956+0000","last_peered":"2023-12-01T17:36:53.343956+0000","last_clean":"2023-12-01T17:36:53.343956+0000","last_became_active":"2023-12-01T17:05:21.296791+0000","last_became_peered":"2023-12-01T17:05:21.296791+0000","last_unstale":"2023-12-01T17:36:53.343956+0000","last_undegraded":"2023-12-01T17:36:53.343956+0000","last_fullsized":"2023-12-01T17:36:53.343956+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:53.342520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:53.342520+0000","last_clean_scrub_stamp":"2023-12-01T17:36:53.342520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T17:54:08.559436+0000","scrub_duration":0.079442730000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":1066,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:45.166465+0000","last_change":"2023-12-01T17:36:45.167142+0000","last_active":"2023-12-01T17:36:45.166465+0000","last_peered":"2023-12-01T17:36:45.166465+0000","last_clean":"2023-12-01T17:36:45.166465+0000","last_became_active":"2023-12-01T17:05:13.265267+0000","last_became_peered":"2023-12-01T17:05:13.265267+0000","last_unstale":"2023-12-01T17:36:45.166465+0000","last_undegraded":"2023-12-01T17:36:45.166465+0000","last_fullsized":"2023-12-01T17:36:45.166465+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:45.165081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:45.165081+0000","last_clean_scrub_stamp":"2023-12-01T17:36:45.165081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T01:07:18.537211+0000","scrub_duration":0.027418776999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.953859+0000","last_change":"2023-12-01T17:36:44.954614+0000","last_active":"2023-12-01T17:36:44.953859+0000","last_peered":"2023-12-01T17:36:44.953859+0000","last_clean":"2023-12-01T17:36:44.953859+0000","last_became_active":"2023-12-01T17:05:38.651540+0000","last_became_peered":"2023-12-01T17:05:38.651540+0000","last_unstale":"2023-12-01T17:36:44.953859+0000","last_undegraded":"2023-12-01T17:36:44.953859+0000","last_fullsized":"2023-12-01T17:36:44.953859+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.952389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.952389+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.952389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:51:40.516240+0000","scrub_duration":0.077883641000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1a","version":"0'0","reported_seq":1071,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:40.853151+0000","last_change":"2023-12-01T17:36:40.853841+0000","last_active":"2023-12-01T17:36:40.853151+0000","last_peered":"2023-12-01T17:36:40.853151+0000","last_clean":"2023-12-01T17:36:40.853151+0000","last_became_active":"2023-12-01T17:05:45.102496+0000","last_became_peered":"2023-12-01T17:05:45.102496+0000","last_unstale":"2023-12-01T17:36:40.853151+0000","last_undegraded":"2023-12-01T17:36:40.853151+0000","last_fullsized":"2023-12-01T17:36:40.853151+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:40.851788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:40.851788+0000","last_clean_scrub_stamp":"2023-12-01T17:36:40.851788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:49:20.220515+0000","scrub_duration":0.042120170999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1b","version":"0'0","reported_seq":1061,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:35.965140+0000","last_change":"2023-12-01T17:36:35.966071+0000","last_active":"2023-12-01T17:36:35.965140+0000","last_peered":"2023-12-01T17:36:35.965140+0000","last_clean":"2023-12-01T17:36:35.965140+0000","last_became_active":"2023-12-01T17:05:21.112377+0000","last_became_peered":"2023-12-01T17:05:21.112377+0000","last_unstale":"2023-12-01T17:36:35.965140+0000","last_undegraded":"2023-12-01T17:36:35.965140+0000","last_fullsized":"2023-12-01T17:36:35.965140+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:35.963469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:35.963469+0000","last_clean_scrub_stamp":"2023-12-01T17:36:35.963469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:36:03.057210+0000","scrub_duration":0.081911166999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:41.123404+0000","last_change":"2023-12-01T17:36:41.123404+0000","last_active":"2023-12-01T17:36:41.123404+0000","last_peered":"2023-12-01T17:36:41.123404+0000","last_clean":"2023-12-01T17:36:41.123404+0000","last_became_active":"2023-12-01T17:05:12.325443+0000","last_became_peered":"2023-12-01T17:05:12.325443+0000","last_unstale":"2023-12-01T17:36:41.123404+0000","last_undegraded":"2023-12-01T17:36:41.123404+0000","last_fullsized":"2023-12-01T17:36:41.123404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:41.121365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:41.121365+0000","last_clean_scrub_stamp":"2023-12-01T17:36:41.121365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:25:47.490775+0000","scrub_duration":0.031595139000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":1062,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:33.940179+0000","last_change":"2023-12-01T17:36:33.940179+0000","last_active":"2023-12-01T17:36:33.940179+0000","last_peered":"2023-12-01T17:36:33.940179+0000","last_clean":"2023-12-01T17:36:33.940179+0000","last_became_active":"2023-12-01T17:05:35.971286+0000","last_became_peered":"2023-12-01T17:05:35.971286+0000","last_unstale":"2023-12-01T17:36:33.940179+0000","last_undegraded":"2023-12-01T17:36:33.940179+0000","last_fullsized":"2023-12-01T17:36:33.940179+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:33.933861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:33.933861+0000","last_clean_scrub_stamp":"2023-12-01T17:36:33.933861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:29:13.183460+0000","scrub_duration":0.050039922000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1e","version":"0'0","reported_seq":1058,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:32.519854+0000","last_change":"2023-12-01T17:36:32.519854+0000","last_active":"2023-12-01T17:36:32.519854+0000","last_peered":"2023-12-01T17:36:32.519854+0000","last_clean":"2023-12-01T17:36:32.519854+0000","last_became_active":"2023-12-01T17:05:33.448448+0000","last_became_peered":"2023-12-01T17:05:33.448448+0000","last_unstale":"2023-12-01T17:36:32.519854+0000","last_undegraded":"2023-12-01T17:36:32.519854+0000","last_fullsized":"2023-12-01T17:36:32.519854+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:32.517436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:32.517436+0000","last_clean_scrub_stamp":"2023-12-01T17:36:32.517436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:57:35.099944+0000","scrub_duration":0.069663173999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:43.961410+0000","last_change":"2023-12-01T17:36:43.962127+0000","last_active":"2023-12-01T17:36:43.961410+0000","last_peered":"2023-12-01T17:36:43.961410+0000","last_clean":"2023-12-01T17:36:43.961410+0000","last_became_active":"2023-12-01T17:05:36.329117+0000","last_became_peered":"2023-12-01T17:05:36.329117+0000","last_unstale":"2023-12-01T17:36:43.961410+0000","last_undegraded":"2023-12-01T17:36:43.961410+0000","last_fullsized":"2023-12-01T17:36:43.961410+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:43.959988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:43.959988+0000","last_clean_scrub_stamp":"2023-12-01T17:36:43.959988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:13:24.416511+0000","scrub_duration":0.068376606000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1c","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.136422+0000","last_change":"2023-12-01T17:36:44.137094+0000","last_active":"2023-12-01T17:36:44.136422+0000","last_peered":"2023-12-01T17:36:44.136422+0000","last_clean":"2023-12-01T17:36:44.136422+0000","last_became_active":"2023-12-01T17:05:11.452081+0000","last_became_peered":"2023-12-01T17:05:11.452081+0000","last_unstale":"2023-12-01T17:36:44.136422+0000","last_undegraded":"2023-12-01T17:36:44.136422+0000","last_fullsized":"2023-12-01T17:36:44.136422+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.135069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.135069+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.135069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:37:10.964902+0000","scrub_duration":0.027982248000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1d","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:47.339466+0000","last_change":"2023-12-01T17:36:47.340125+0000","last_active":"2023-12-01T17:36:47.339466+0000","last_peered":"2023-12-01T17:36:47.339466+0000","last_clean":"2023-12-01T17:36:47.339466+0000","last_became_active":"2023-12-01T17:05:36.185378+0000","last_became_peered":"2023-12-01T17:05:36.185378+0000","last_unstale":"2023-12-01T17:36:47.339466+0000","last_undegraded":"2023-12-01T17:36:47.339466+0000","last_fullsized":"2023-12-01T17:36:47.339466+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:47.338105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:47.338105+0000","last_clean_scrub_stamp":"2023-12-01T17:36:47.338105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:50:41.146788+0000","scrub_duration":0.064980219000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"1099'78","reported_seq":1174,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:34.048890+0000","last_change":"2023-12-01T17:36:34.050160+0000","last_active":"2023-12-01T17:36:34.048890+0000","last_peered":"2023-12-01T17:36:34.048890+0000","last_clean":"2023-12-01T17:36:34.048890+0000","last_became_active":"2023-12-01T17:03:54.856410+0000","last_became_peered":"2023-12-01T17:03:54.856410+0000","last_unstale":"2023-12-01T17:36:34.048890+0000","last_undegraded":"2023-12-01T17:36:34.048890+0000","last_fullsized":"2023-12-01T17:36:34.048890+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'78","last_scrub_stamp":"2023-12-01T17:36:34.044757+0000","last_deep_scrub":"1099'78","last_deep_scrub_stamp":"2023-12-01T17:36:34.044757+0000","last_clean_scrub_stamp":"2023-12-01T17:36:34.044757+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:16:14.393778+0000","scrub_duration":0.143069317,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":1062,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:38.933961+0000","last_change":"2023-12-01T17:36:38.934686+0000","last_active":"2023-12-01T17:36:38.933961+0000","last_peered":"2023-12-01T17:36:38.933961+0000","last_clean":"2023-12-01T17:36:38.933961+0000","last_became_active":"2023-12-01T17:04:01.489613+0000","last_became_peered":"2023-12-01T17:04:01.489613+0000","last_unstale":"2023-12-01T17:36:38.933961+0000","last_undegraded":"2023-12-01T17:36:38.933961+0000","last_fullsized":"2023-12-01T17:36:38.933961+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:38.932452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:38.932452+0000","last_clean_scrub_stamp":"2023-12-01T17:36:38.932452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:21:52.995129+0000","scrub_duration":0.044391881000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":1058,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:46.149295+0000","last_change":"2023-12-01T17:36:46.150030+0000","last_active":"2023-12-01T17:36:46.149295+0000","last_peered":"2023-12-01T17:36:46.149295+0000","last_clean":"2023-12-01T17:36:46.149295+0000","last_became_active":"2023-12-01T17:05:15.659631+0000","last_became_peered":"2023-12-01T17:05:15.659631+0000","last_unstale":"2023-12-01T17:36:46.149295+0000","last_undegraded":"2023-12-01T17:36:46.149295+0000","last_fullsized":"2023-12-01T17:36:46.149295+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:46.147862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:46.147862+0000","last_clean_scrub_stamp":"2023-12-01T17:36:46.147862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T04:49:27.130671+0000","scrub_duration":0.076369635000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:35.850307+0000","last_change":"2023-12-01T17:36:35.851277+0000","last_active":"2023-12-01T17:36:35.850307+0000","last_peered":"2023-12-01T17:36:35.850307+0000","last_clean":"2023-12-01T17:36:35.850307+0000","last_became_active":"2023-12-01T17:04:01.117201+0000","last_became_peered":"2023-12-01T17:04:01.117201+0000","last_unstale":"2023-12-01T17:36:35.850307+0000","last_undegraded":"2023-12-01T17:36:35.850307+0000","last_fullsized":"2023-12-01T17:36:35.850307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:35.848834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:35.848834+0000","last_clean_scrub_stamp":"2023-12-01T17:36:35.848834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:13:37.157785+0000","scrub_duration":0.034576843000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.14","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:39.047159+0000","last_change":"2023-12-01T17:36:39.047877+0000","last_active":"2023-12-01T17:36:39.047159+0000","last_peered":"2023-12-01T17:36:39.047159+0000","last_clean":"2023-12-01T17:36:39.047159+0000","last_became_active":"2023-12-01T17:05:11.329382+0000","last_became_peered":"2023-12-01T17:05:11.329382+0000","last_unstale":"2023-12-01T17:36:39.047159+0000","last_undegraded":"2023-12-01T17:36:39.047159+0000","last_fullsized":"2023-12-01T17:36:39.047159+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:39.045715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:39.045715+0000","last_clean_scrub_stamp":"2023-12-01T17:36:39.045715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:19:56.059217+0000","scrub_duration":0.032303137000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1070,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.338016+0000","last_change":"2023-12-01T17:36:44.338677+0000","last_active":"2023-12-01T17:36:44.338016+0000","last_peered":"2023-12-01T17:36:44.338016+0000","last_clean":"2023-12-01T17:36:44.338016+0000","last_became_active":"2023-12-01T17:04:01.076736+0000","last_became_peered":"2023-12-01T17:04:01.076736+0000","last_unstale":"2023-12-01T17:36:44.338016+0000","last_undegraded":"2023-12-01T17:36:44.338016+0000","last_fullsized":"2023-12-01T17:36:44.338016+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.336665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.336665+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.336665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:01:19.263355+0000","scrub_duration":0.040984052,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.15","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:49.138747+0000","last_change":"2023-12-01T17:36:49.139443+0000","last_active":"2023-12-01T17:36:49.138747+0000","last_peered":"2023-12-01T17:36:49.138747+0000","last_clean":"2023-12-01T17:36:49.138747+0000","last_became_active":"2023-12-01T17:05:16.481672+0000","last_became_peered":"2023-12-01T17:05:16.481672+0000","last_unstale":"2023-12-01T17:36:49.138747+0000","last_undegraded":"2023-12-01T17:36:49.138747+0000","last_fullsized":"2023-12-01T17:36:49.138747+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:49.137364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:49.137364+0000","last_clean_scrub_stamp":"2023-12-01T17:36:49.137364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:28:04.767796+0000","scrub_duration":0.050542482999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:43.972358+0000","last_change":"2023-12-01T17:36:43.973083+0000","last_active":"2023-12-01T17:36:43.972358+0000","last_peered":"2023-12-01T17:36:43.972358+0000","last_clean":"2023-12-01T17:36:43.972358+0000","last_became_active":"2023-12-01T17:04:01.424410+0000","last_became_peered":"2023-12-01T17:04:01.424410+0000","last_unstale":"2023-12-01T17:36:43.972358+0000","last_undegraded":"2023-12-01T17:36:43.972358+0000","last_fullsized":"2023-12-01T17:36:43.972358+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:43.970938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:43.970938+0000","last_clean_scrub_stamp":"2023-12-01T17:36:43.970938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:30:06.736333+0000","scrub_duration":0.035023907,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":1064,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:50.267325+0000","last_change":"2023-12-01T17:36:50.267970+0000","last_active":"2023-12-01T17:36:50.267325+0000","last_peered":"2023-12-01T17:36:50.267325+0000","last_clean":"2023-12-01T17:36:50.267325+0000","last_became_active":"2023-12-01T17:05:14.948506+0000","last_became_peered":"2023-12-01T17:05:14.948506+0000","last_unstale":"2023-12-01T17:36:50.267325+0000","last_undegraded":"2023-12-01T17:36:50.267325+0000","last_fullsized":"2023-12-01T17:36:50.267325+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:50.265821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:50.265821+0000","last_clean_scrub_stamp":"2023-12-01T17:36:50.265821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:30:58.709148+0000","scrub_duration":0.039070454999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1069,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:37.179004+0000","last_change":"2023-12-01T17:36:37.179968+0000","last_active":"2023-12-01T17:36:37.179004+0000","last_peered":"2023-12-01T17:36:37.179004+0000","last_clean":"2023-12-01T17:36:37.179004+0000","last_became_active":"2023-12-01T17:04:01.438712+0000","last_became_peered":"2023-12-01T17:04:01.438712+0000","last_unstale":"2023-12-01T17:36:37.179004+0000","last_undegraded":"2023-12-01T17:36:37.179004+0000","last_fullsized":"2023-12-01T17:36:37.179004+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:37.177549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:37.177549+0000","last_clean_scrub_stamp":"2023-12-01T17:36:37.177549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:55:33.275441+0000","scrub_duration":0.035436190999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:37.013581+0000","last_change":"2023-12-01T17:36:37.014317+0000","last_active":"2023-12-01T17:36:37.013581+0000","last_peered":"2023-12-01T17:36:37.013581+0000","last_clean":"2023-12-01T17:36:37.013581+0000","last_became_active":"2023-12-01T17:05:11.693016+0000","last_became_peered":"2023-12-01T17:05:11.693016+0000","last_unstale":"2023-12-01T17:36:37.013581+0000","last_undegraded":"2023-12-01T17:36:37.013581+0000","last_fullsized":"2023-12-01T17:36:37.013581+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:37.012131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:37.012131+0000","last_clean_scrub_stamp":"2023-12-01T17:36:37.012131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:18:23.046760+0000","scrub_duration":0.033385561000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1063,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:49.902910+0000","last_change":"2023-12-01T17:36:49.903621+0000","last_active":"2023-12-01T17:36:49.902910+0000","last_peered":"2023-12-01T17:36:49.902910+0000","last_clean":"2023-12-01T17:36:49.902910+0000","last_became_active":"2023-12-01T17:04:01.195094+0000","last_became_peered":"2023-12-01T17:04:01.195094+0000","last_unstale":"2023-12-01T17:36:49.902910+0000","last_undegraded":"2023-12-01T17:36:49.902910+0000","last_fullsized":"2023-12-01T17:36:49.902910+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:49.901478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:49.901478+0000","last_clean_scrub_stamp":"2023-12-01T17:36:49.901478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:09:38.122593+0000","scrub_duration":0.037209064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":1075,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:42.514285+0000","last_change":"2023-12-01T17:36:42.514948+0000","last_active":"2023-12-01T17:36:42.514285+0000","last_peered":"2023-12-01T17:36:42.514285+0000","last_clean":"2023-12-01T17:36:42.514285+0000","last_became_active":"2023-12-01T17:05:12.768446+0000","last_became_peered":"2023-12-01T17:05:12.768446+0000","last_unstale":"2023-12-01T17:36:42.514285+0000","last_undegraded":"2023-12-01T17:36:42.514285+0000","last_fullsized":"2023-12-01T17:36:42.514285+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:42.512983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:42.512983+0000","last_clean_scrub_stamp":"2023-12-01T17:36:42.512983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:54:53.202156+0000","scrub_duration":0.051052442000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1077,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:37.836360+0000","last_change":"2023-12-01T17:36:37.837000+0000","last_active":"2023-12-01T17:36:37.836360+0000","last_peered":"2023-12-01T17:36:37.836360+0000","last_clean":"2023-12-01T17:36:37.836360+0000","last_became_active":"2023-12-01T17:04:01.508824+0000","last_became_peered":"2023-12-01T17:04:01.508824+0000","last_unstale":"2023-12-01T17:36:37.836360+0000","last_undegraded":"2023-12-01T17:36:37.836360+0000","last_fullsized":"2023-12-01T17:36:37.836360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:37.835002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:37.835002+0000","last_clean_scrub_stamp":"2023-12-01T17:36:37.835002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:52:23.931657+0000","scrub_duration":0.033273367999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":1066,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:45.341446+0000","last_change":"2023-12-01T17:36:45.342107+0000","last_active":"2023-12-01T17:36:45.341446+0000","last_peered":"2023-12-01T17:36:45.341446+0000","last_clean":"2023-12-01T17:36:45.341446+0000","last_became_active":"2023-12-01T17:05:24.924547+0000","last_became_peered":"2023-12-01T17:05:24.924547+0000","last_unstale":"2023-12-01T17:36:45.341446+0000","last_undegraded":"2023-12-01T17:36:45.341446+0000","last_fullsized":"2023-12-01T17:36:45.341446+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:45.340092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:45.340092+0000","last_clean_scrub_stamp":"2023-12-01T17:36:45.340092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T21:42:17.355162+0000","scrub_duration":0.034252206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:44.952033+0000","last_change":"2023-12-01T17:36:44.952769+0000","last_active":"2023-12-01T17:36:44.952033+0000","last_peered":"2023-12-01T17:36:44.952033+0000","last_clean":"2023-12-01T17:36:44.952033+0000","last_became_active":"2023-12-01T17:04:01.467992+0000","last_became_peered":"2023-12-01T17:04:01.467992+0000","last_unstale":"2023-12-01T17:36:44.952033+0000","last_undegraded":"2023-12-01T17:36:44.952033+0000","last_fullsized":"2023-12-01T17:36:44.952033+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:44.948272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:44.948272+0000","last_clean_scrub_stamp":"2023-12-01T17:36:44.948272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:34:41.622283+0000","scrub_duration":0.049061754999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"0'0","reported_seq":1072,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:38.895535+0000","last_change":"2023-12-01T17:36:38.896418+0000","last_active":"2023-12-01T17:36:38.895535+0000","last_peered":"2023-12-01T17:36:38.895535+0000","last_clean":"2023-12-01T17:36:38.895535+0000","last_became_active":"2023-12-01T17:05:28.748541+0000","last_became_peered":"2023-12-01T17:05:28.748541+0000","last_unstale":"2023-12-01T17:36:38.895535+0000","last_undegraded":"2023-12-01T17:36:38.895535+0000","last_fullsized":"2023-12-01T17:36:38.895535+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:38.894158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:38.894158+0000","last_clean_scrub_stamp":"2023-12-01T17:36:38.894158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:49:49.640669+0000","scrub_duration":0.076043471000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1","version":"1099'8","reported_seq":1078,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:47.847022+0000","last_change":"2023-12-01T17:36:47.848174+0000","last_active":"2023-12-01T17:36:47.847022+0000","last_peered":"2023-12-01T17:36:47.847022+0000","last_clean":"2023-12-01T17:36:47.847022+0000","last_became_active":"2023-12-01T17:05:38.775069+0000","last_became_peered":"2023-12-01T17:05:38.775069+0000","last_unstale":"2023-12-01T17:36:47.847022+0000","last_undegraded":"2023-12-01T17:36:47.847022+0000","last_fullsized":"2023-12-01T17:36:47.847022+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"1099'8","last_scrub_stamp":"2023-12-01T17:36:47.845650+0000","last_deep_scrub":"1099'8","last_deep_scrub_stamp":"2023-12-01T17:36:47.845650+0000","last_clean_scrub_stamp":"2023-12-01T17:36:47.845650+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T20:06:25.294586+0000","scrub_duration":0.084520799999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":1065,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:34.895082+0000","last_change":"2023-12-01T17:36:34.895800+0000","last_active":"2023-12-01T17:36:34.895082+0000","last_peered":"2023-12-01T17:36:34.895082+0000","last_clean":"2023-12-01T17:36:34.895082+0000","last_became_active":"2023-12-01T17:05:17.447969+0000","last_became_peered":"2023-12-01T17:05:17.447969+0000","last_unstale":"2023-12-01T17:36:34.895082+0000","last_undegraded":"2023-12-01T17:36:34.895082+0000","last_fullsized":"2023-12-01T17:36:34.895082+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:34.893658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:34.893658+0000","last_clean_scrub_stamp":"2023-12-01T17:36:34.893658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:27:39.596435+0000","scrub_duration":0.034514954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":1062,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:38.080186+0000","last_change":"2023-12-01T17:36:38.080912+0000","last_active":"2023-12-01T17:36:38.080186+0000","last_peered":"2023-12-01T17:36:38.080186+0000","last_clean":"2023-12-01T17:36:38.080186+0000","last_became_active":"2023-12-01T17:05:12.334114+0000","last_became_peered":"2023-12-01T17:05:12.334114+0000","last_unstale":"2023-12-01T17:36:38.080186+0000","last_undegraded":"2023-12-01T17:36:38.080186+0000","last_fullsized":"2023-12-01T17:36:38.080186+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:38.076259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:38.076259+0000","last_clean_scrub_stamp":"2023-12-01T17:36:38.076259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:35:13.808207+0000","scrub_duration":0.069810686999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":1077,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:43.482858+0000","last_change":"2023-12-01T17:36:43.483544+0000","last_active":"2023-12-01T17:36:43.482858+0000","last_peered":"2023-12-01T17:36:43.482858+0000","last_clean":"2023-12-01T17:36:43.482858+0000","last_became_active":"2023-12-01T17:05:39.340484+0000","last_became_peered":"2023-12-01T17:05:39.340484+0000","last_unstale":"2023-12-01T17:36:43.482858+0000","last_undegraded":"2023-12-01T17:36:43.482858+0000","last_fullsized":"2023-12-01T17:36:43.482858+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:43.481522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:43.481522+0000","last_clean_scrub_stamp":"2023-12-01T17:36:43.481522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T02:00:59.719215+0000","scrub_duration":0.034362386000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"22.d","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:40.021477+0000","last_change":"2023-12-01T17:36:40.021477+0000","last_active":"2023-12-01T17:36:40.021477+0000","last_peered":"2023-12-01T17:36:40.021477+0000","last_clean":"2023-12-01T17:36:40.021477+0000","last_became_active":"2023-12-01T17:05:58.772230+0000","last_became_peered":"2023-12-01T17:05:58.772230+0000","last_unstale":"2023-12-01T17:36:40.021477+0000","last_undegraded":"2023-12-01T17:36:40.021477+0000","last_fullsized":"2023-12-01T17:36:40.021477+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:40.019346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:40.019346+0000","last_clean_scrub_stamp":"2023-12-01T17:36:40.019346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T18:34:11.691955+0000","scrub_duration":0.049893717999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":1058,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:51.958138+0000","last_change":"2023-12-01T17:36:51.958865+0000","last_active":"2023-12-01T17:36:51.958138+0000","last_peered":"2023-12-01T17:36:51.958138+0000","last_clean":"2023-12-01T17:36:51.958138+0000","last_became_active":"2023-12-01T17:05:11.209283+0000","last_became_peered":"2023-12-01T17:05:11.209283+0000","last_unstale":"2023-12-01T17:36:51.958138+0000","last_undegraded":"2023-12-01T17:36:51.958138+0000","last_fullsized":"2023-12-01T17:36:51.958138+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:51.954556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:51.954556+0000","last_clean_scrub_stamp":"2023-12-01T17:36:51.954556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T05:15:04.690916+0000","scrub_duration":0.046634676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.f","version":"0'0","reported_seq":1060,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:45.122583+0000","last_change":"2023-12-01T17:36:45.122583+0000","last_active":"2023-12-01T17:36:45.122583+0000","last_peered":"2023-12-01T17:36:45.122583+0000","last_clean":"2023-12-01T17:36:45.122583+0000","last_became_active":"2023-12-01T17:05:17.215461+0000","last_became_peered":"2023-12-01T17:05:17.215461+0000","last_unstale":"2023-12-01T17:36:45.122583+0000","last_undegraded":"2023-12-01T17:36:45.122583+0000","last_fullsized":"2023-12-01T17:36:45.122583+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:45.119993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:45.119993+0000","last_clean_scrub_stamp":"2023-12-01T17:36:45.119993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:37:06.008536+0000","scrub_duration":0.094950128999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.e","version":"0'0","reported_seq":1056,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:36.547455+0000","last_change":"2023-12-01T17:36:36.547455+0000","last_active":"2023-12-01T17:36:36.547455+0000","last_peered":"2023-12-01T17:36:36.547455+0000","last_clean":"2023-12-01T17:36:36.547455+0000","last_became_active":"2023-12-01T17:05:21.668214+0000","last_became_peered":"2023-12-01T17:05:21.668214+0000","last_unstale":"2023-12-01T17:36:36.547455+0000","last_undegraded":"2023-12-01T17:36:36.547455+0000","last_fullsized":"2023-12-01T17:36:36.547455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:36.542404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:36.542404+0000","last_clean_scrub_stamp":"2023-12-01T17:36:36.542404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T23:29:17.880198+0000","scrub_duration":0.041798604000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":1073,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:42.788666+0000","last_change":"2023-12-01T17:36:42.789790+0000","last_active":"2023-12-01T17:36:42.788666+0000","last_peered":"2023-12-01T17:36:42.788666+0000","last_clean":"2023-12-01T17:36:42.788666+0000","last_became_active":"2023-12-01T17:05:46.016610+0000","last_became_peered":"2023-12-01T17:05:46.016610+0000","last_unstale":"2023-12-01T17:36:42.788666+0000","last_undegraded":"2023-12-01T17:36:42.788666+0000","last_fullsized":"2023-12-01T17:36:42.788666+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:42.786419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:42.786419+0000","last_clean_scrub_stamp":"2023-12-01T17:36:42.786419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-03T03:14:29.992685+0000","scrub_duration":0.054486035000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":1059,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:50.899559+0000","last_change":"2023-12-01T17:36:50.900297+0000","last_active":"2023-12-01T17:36:50.899559+0000","last_peered":"2023-12-01T17:36:50.899559+0000","last_clean":"2023-12-01T17:36:50.899559+0000","last_became_active":"2023-12-01T17:05:37.358444+0000","last_became_peered":"2023-12-01T17:05:37.358444+0000","last_unstale":"2023-12-01T17:36:50.899559+0000","last_undegraded":"2023-12-01T17:36:50.899559+0000","last_fullsized":"2023-12-01T17:36:50.899559+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:50.895912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:50.895912+0000","last_clean_scrub_stamp":"2023-12-01T17:36:50.895912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:10:33.121163+0000","scrub_duration":0.031742754999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.b","version":"0'0","reported_seq":1065,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:48.133542+0000","last_change":"2023-12-01T17:36:48.134215+0000","last_active":"2023-12-01T17:36:48.133542+0000","last_peered":"2023-12-01T17:36:48.133542+0000","last_clean":"2023-12-01T17:36:48.133542+0000","last_became_active":"2023-12-01T17:05:13.499129+0000","last_became_peered":"2023-12-01T17:05:13.499129+0000","last_unstale":"2023-12-01T17:36:48.133542+0000","last_undegraded":"2023-12-01T17:36:48.133542+0000","last_fullsized":"2023-12-01T17:36:48.133542+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:48.132123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:48.132123+0000","last_clean_scrub_stamp":"2023-12-01T17:36:48.132123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T19:07:01.107754+0000","scrub_duration":0.083408903000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":1059,"reported_epoch":1099,"state":"active+clean","last_fresh":"2023-12-01T17:36:52.941682+0000","last_change":"2023-12-01T17:36:52.942402+0000","last_active":"2023-12-01T17:36:52.941682+0000","last_peered":"2023-12-01T17:36:52.941682+0000","last_clean":"2023-12-01T17:36:52.941682+0000","last_became_active":"2023-12-01T17:05:37.740514+0000","last_became_peered":"2023-12-01T17:05:37.740514+0000","last_unstale":"2023-12-01T17:36:52.941682+0000","last_undegraded":"2023-12-01T17:36:52.941682+0000","last_fullsized":"2023-12-01T17:36:52.941682+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-01T17:36:52.940307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-01T17:36:52.940307+0000","last_clean_scrub_stamp":"2023-12-01T17:36:52.940307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-02T22:02:53.085671+0000","scrub_duration":0.043691500000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":1,"up_from":17,"seq":73014444431,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":70172,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301668,"statfs":{"total":96636764160,"available":96564908032,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":20.553999999999998,"5min":12.125999999999999,"15min":15.801},"min":{"1min":6.5620000000000003,"5min":4.5899999999999999,"15min":4.5899999999999999},"max":{"1min":127.741,"5min":127.741,"15min":150.279},"last":5.75},{"interface":"front","average":{"1min":29.547000000000001,"5min":14.099,"15min":17.922999999999998},"min":{"1min":8.5419999999999998,"5min":4.7229999999999999,"15min":4.7229999999999999},"max":{"1min":184.30600000000001,"5min":184.30600000000001,"15min":184.30600000000001},"last":11.380000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":18.321999999999999,"5min":12.057,"15min":16.654},"min":{"1min":6.5709999999999997,"5min":5.4610000000000003,"15min":5.4610000000000003},"max":{"1min":99.156000000000006,"5min":99.156000000000006,"15min":208.18199999999999},"last":7.9660000000000002},{"interface":"front","average":{"1min":34.377000000000002,"5min":16.986999999999998,"15min":19.702000000000002},"min":{"1min":7.9260000000000002,"5min":6.8769999999999998,"15min":6.2030000000000003},"max":{"1min":216.785,"5min":216.785,"15min":216.785},"last":13.776999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":28.09,"5min":16.707999999999998,"15min":21.498000000000001},"min":{"1min":8.9960000000000004,"5min":7.8550000000000004,"15min":7.8550000000000004},"max":{"1min":243.88999999999999,"5min":243.88999999999999,"15min":276.07400000000001},"last":12.054},{"interface":"front","average":{"1min":30.187000000000001,"5min":19.353999999999999,"15min":24.324000000000002},"min":{"1min":13.336,"5min":10.973000000000001,"15min":7.2649999999999997},"max":{"1min":239.40000000000001,"5min":239.40000000000001,"15min":252.43299999999999},"last":14.407}]},{"osd":4,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":31.789000000000001,"5min":16.448,"15min":17.77},"min":{"1min":8.0429999999999993,"5min":6.7400000000000002,"15min":6.7400000000000002},"max":{"1min":230.511,"5min":230.511,"15min":230.511},"last":18.105},{"interface":"front","average":{"1min":33.975000000000001,"5min":17.349,"15min":21.777999999999999},"min":{"1min":7.6379999999999999,"5min":7.3979999999999997,"15min":7.3979999999999997},"max":{"1min":241.28100000000001,"5min":241.28100000000001,"15min":241.28100000000001},"last":8.9190000000000005}]},{"osd":5,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":27.834,"5min":16.713999999999999,"15min":18.786000000000001},"min":{"1min":6.859,"5min":6.859,"15min":6.149},"max":{"1min":191.78700000000001,"5min":191.78700000000001,"15min":191.78700000000001},"last":17.059000000000001},{"interface":"front","average":{"1min":17.850999999999999,"5min":15.477,"15min":22.440999999999999},"min":{"1min":7.3769999999999998,"5min":7.3769999999999998,"15min":7.0270000000000001},"max":{"1min":111.208,"5min":111.208,"15min":311.13299999999998},"last":13.029999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":22.614999999999998,"5min":17.094999999999999,"15min":18.754999999999999},"min":{"1min":11.262,"5min":10.199,"15min":7.0960000000000001},"max":{"1min":94.787999999999997,"5min":94.787999999999997,"15min":99.623000000000005},"last":11.1},{"interface":"front","average":{"1min":26.064,"5min":17.611000000000001,"15min":21.722999999999999},"min":{"1min":8.6600000000000001,"5min":8.6080000000000005,"15min":7.9909999999999997},"max":{"1min":153.83799999999999,"5min":153.83799999999999,"15min":186.11000000000001},"last":12.369}]},{"osd":7,"last update":"Fri Dec 1 17:36:18 2023","interfaces":[{"interface":"back","average":{"1min":35.716000000000001,"5min":20.347000000000001,"15min":24.768999999999998},"min":{"1min":12.414999999999999,"5min":9.4819999999999993,"15min":9.1150000000000002},"max":{"1min":207.64599999999999,"5min":207.64599999999999,"15min":207.64599999999999},"last":19.074000000000002},{"interface":"front","average":{"1min":23.535,"5min":17.969999999999999,"15min":21.334},"min":{"1min":11.978999999999999,"5min":9.4469999999999992,"15min":9.0069999999999997},"max":{"1min":74.777000000000001,"5min":74.777000000000001,"15min":148.89599999999999},"last":16.323}]}]},{"osd":3,"up_from":16,"seq":68719477135,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69600,"kb_used_data":6488,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302240,"statfs":{"total":96636764160,"available":96565493760,"internally_reserved":0,"allocated":6643712,"data_stored":3599218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":38.427,"5min":15.26,"15min":18.015000000000001},"min":{"1min":6.8440000000000003,"5min":5.5540000000000003,"15min":5.2539999999999996},"max":{"1min":180.16,"5min":180.16,"15min":181.94200000000001},"last":9.4030000000000005},{"interface":"front","average":{"1min":28.507000000000001,"5min":13.943,"15min":18.838000000000001},"min":{"1min":4.915,"5min":4.915,"15min":4.915},"max":{"1min":107.5,"5min":107.5,"15min":427.36500000000001},"last":6.6050000000000004}]},{"osd":1,"last update":"Fri Dec 1 17:36:20 2023","interfaces":[{"interface":"back","average":{"1min":30.577999999999999,"5min":15.76,"15min":20.832999999999998},"min":{"1min":8.1419999999999995,"5min":7.585,"15min":6.4039999999999999},"max":{"1min":146.48099999999999,"5min":146.48099999999999,"15min":227.309},"last":10.760999999999999},{"interface":"front","average":{"1min":33.158000000000001,"5min":15.33,"15min":19.213999999999999},"min":{"1min":8.2870000000000008,"5min":6.9850000000000003,"15min":5.827},"max":{"1min":223.179,"5min":223.179,"15min":421.40600000000001},"last":13.282999999999999}]},{"osd":2,"last update":"Fri Dec 1 17:37:06 2023","interfaces":[{"interface":"back","average":{"1min":19.593,"5min":14.84,"15min":21.989999999999998},"min":{"1min":8.282,"5min":7.3029999999999999,"15min":7.3029999999999999},"max":{"1min":121.88200000000001,"5min":121.88200000000001,"15min":451.97899999999998},"last":8.7880000000000003},{"interface":"front","average":{"1min":15.811999999999999,"5min":16.626999999999999,"15min":23.327999999999999},"min":{"1min":8.4600000000000009,"5min":6.7190000000000003,"15min":6.5170000000000003},"max":{"1min":39.487000000000002,"5min":223.84,"15min":384.58699999999999},"last":15.445}]},{"osd":4,"last update":"Fri Dec 1 17:37:06 2023","interfaces":[{"interface":"back","average":{"1min":16.207000000000001,"5min":18.192,"15min":21.521999999999998},"min":{"1min":9.9760000000000009,"5min":8.5640000000000001,"15min":6.5330000000000004},"max":{"1min":28.710999999999999,"5min":244.989,"15min":424.09500000000003},"last":16.885000000000002},{"interface":"front","average":{"1min":18.315000000000001,"5min":18.077999999999999,"15min":20.782},"min":{"1min":8.7550000000000008,"5min":7.9870000000000001,"15min":7.0049999999999999},"max":{"1min":80.042000000000002,"5min":159.78200000000001,"15min":202.70500000000001},"last":10.074}]},{"osd":5,"last update":"Fri Dec 1 17:37:06 2023","interfaces":[{"interface":"back","average":{"1min":35.292000000000002,"5min":23.164999999999999,"15min":25.815000000000001},"min":{"1min":10.718,"5min":10.481999999999999,"15min":9.0860000000000003},"max":{"1min":158.989,"5min":228.96700000000001,"15min":424.80099999999999},"last":17.526},{"interface":"front","average":{"1min":34.603999999999999,"5min":19.786999999999999,"15min":21.919},"min":{"1min":10.884,"5min":8.0359999999999996,"15min":7.1710000000000003},"max":{"1min":165.26900000000001,"5min":165.26900000000001,"15min":231.65199999999999},"last":18.199999999999999}]},{"osd":6,"last update":"Fri Dec 1 17:37:06 2023","interfaces":[{"interface":"back","average":{"1min":26.515999999999998,"5min":19.934999999999999,"15min":21.986000000000001},"min":{"1min":10.311999999999999,"5min":8.2859999999999996,"15min":7.9340000000000002},"max":{"1min":166.06399999999999,"5min":166.06399999999999,"15min":203.99100000000001},"last":11.349},{"interface":"front","average":{"1min":32.845999999999997,"5min":23.126000000000001,"15min":26.356000000000002},"min":{"1min":12.103,"5min":11.444000000000001,"15min":10.978},"max":{"1min":166.833,"5min":166.833,"15min":203.34100000000001},"last":16.245999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:37:06 2023","interfaces":[{"interface":"back","average":{"1min":21.803999999999998,"5min":18.352,"15min":22.725000000000001},"min":{"1min":9.5670000000000002,"5min":7.9589999999999996,"15min":7.0259999999999998},"max":{"1min":69.914000000000001,"5min":105.342,"15min":196.05699999999999},"last":11.955},{"interface":"front","average":{"1min":24.065999999999999,"5min":20.510000000000002,"15min":25.488},"min":{"1min":13.141999999999999,"5min":11.186,"15min":10.492000000000001},"max":{"1min":71.027000000000001,"5min":102.29000000000001,"15min":236.84700000000001},"last":12.574}]}]},{"osd":2,"up_from":16,"seq":68719477133,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69604,"kb_used_data":6492,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302236,"statfs":{"total":96636764160,"available":96565489664,"internally_reserved":0,"allocated":6647808,"data_stored":3599223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":20.748999999999999,"5min":16.390999999999998,"15min":20.75},"min":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":4.6059999999999999},"max":{"1min":93.664000000000001,"5min":169.971,"15min":231.76499999999999},"last":14.664},{"interface":"front","average":{"1min":20.614000000000001,"5min":16.652000000000001,"15min":20.068000000000001},"min":{"1min":7.7880000000000003,"5min":5.25,"15min":4.8810000000000002},"max":{"1min":94.558000000000007,"5min":150.74199999999999,"15min":222.62},"last":8.3490000000000002}]},{"osd":1,"last update":"Fri Dec 1 17:37:06 2023","interfaces":[{"interface":"back","average":{"1min":25.206,"5min":15.845000000000001,"15min":20.888000000000002},"min":{"1min":6.2649999999999997,"5min":6.1589999999999998,"15min":6.1589999999999998},"max":{"1min":124.76000000000001,"5min":182.83600000000001,"15min":220.96700000000001},"last":10.866},{"interface":"front","average":{"1min":27.411000000000001,"5min":19.417000000000002,"15min":24.103000000000002},"min":{"1min":7.8879999999999999,"5min":7.8879999999999999,"15min":7.8639999999999999},"max":{"1min":122.78400000000001,"5min":196.886,"15min":213.27799999999999},"last":13.119}]},{"osd":3,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":18.446999999999999,"5min":18.890000000000001,"15min":22.745999999999999},"min":{"1min":8.2360000000000007,"5min":7.0609999999999999,"15min":6.734},"max":{"1min":77.816000000000003,"5min":177.04900000000001,"15min":219.51300000000001},"last":15.646000000000001},{"interface":"front","average":{"1min":22.061,"5min":21.041,"15min":24.803000000000001},"min":{"1min":9.2629999999999999,"5min":7.6740000000000004,"15min":6.9900000000000002},"max":{"1min":119.816,"5min":204.721,"15min":204.721},"last":13.792999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":27.114999999999998,"5min":19.007999999999999,"15min":21.718},"min":{"1min":8.6340000000000003,"5min":7.2060000000000004,"15min":6.6799999999999997},"max":{"1min":152.12799999999999,"5min":180.369,"15min":251.745},"last":6.9429999999999996},{"interface":"front","average":{"1min":28.859000000000002,"5min":21.920999999999999,"15min":25.196000000000002},"min":{"1min":10.750999999999999,"5min":10.202999999999999,"15min":7.5720000000000001},"max":{"1min":160.38300000000001,"5min":170.637,"15min":287.32100000000003},"last":20.248000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":19.798999999999999,"5min":19.488,"15min":24.375},"min":{"1min":10.068,"5min":8.4499999999999993,"15min":8.0020000000000007},"max":{"1min":91.971000000000004,"5min":206.56700000000001,"15min":210.012},"last":12.172000000000001},{"interface":"front","average":{"1min":20.013000000000002,"5min":20.241,"15min":25.216000000000001},"min":{"1min":10.693,"5min":8.1059999999999999,"15min":8.1059999999999999},"max":{"1min":83.409000000000006,"5min":232.91900000000001,"15min":232.91900000000001},"last":17.192}]},{"osd":6,"last update":"Fri Dec 1 17:37:06 2023","interfaces":[{"interface":"back","average":{"1min":29.268000000000001,"5min":22.370999999999999,"15min":24.869},"min":{"1min":10.603999999999999,"5min":8.8710000000000004,"15min":8.2699999999999996},"max":{"1min":117.06699999999999,"5min":182.07400000000001,"15min":237.21899999999999},"last":18.088000000000001},{"interface":"front","average":{"1min":32.502000000000002,"5min":23.507999999999999,"15min":28.456},"min":{"1min":12.826000000000001,"5min":9.5960000000000001,"15min":9.5649999999999995},"max":{"1min":119.593,"5min":185.90299999999999,"15min":241.92699999999999},"last":19.257999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:36:50 2023","interfaces":[{"interface":"back","average":{"1min":34.993000000000002,"5min":20.754000000000001,"15min":24.891999999999999},"min":{"1min":8.9269999999999996,"5min":8.7940000000000005,"15min":8.7940000000000005},"max":{"1min":181.03,"5min":181.03,"15min":341.73000000000002},"last":11.497999999999999},{"interface":"front","average":{"1min":38.670000000000002,"5min":23.158999999999999,"15min":28.199999999999999},"min":{"1min":11.782,"5min":10.853,"15min":10.853},"max":{"1min":184.22999999999999,"5min":184.22999999999999,"15min":343.17399999999998},"last":21.606999999999999}]}]},{"osd":0,"up_from":15,"seq":64424509838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69972,"kb_used_data":6860,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94301868,"statfs":{"total":96636764160,"available":96565112832,"internally_reserved":0,"allocated":7024640,"data_stored":3846006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Dec 1 17:36:15 2023","interfaces":[{"interface":"back","average":{"1min":11.167,"5min":10.766999999999999,"15min":17.079000000000001},"min":{"1min":5.5579999999999998,"5min":5.5579999999999998,"15min":4.3550000000000004},"max":{"1min":33.322000000000003,"5min":33.322000000000003,"15min":210.43299999999999},"last":8.2170000000000005},{"interface":"front","average":{"1min":12.661,"5min":10.788,"15min":18.704999999999998},"min":{"1min":6.242,"5min":6.0750000000000002,"15min":5.2590000000000003},"max":{"1min":44.020000000000003,"5min":44.020000000000003,"15min":257.30500000000001},"last":8.9719999999999995}]},{"osd":2,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.808,"5min":14.16,"15min":19.648},"min":{"1min":8.2010000000000005,"5min":7.4740000000000002,"15min":6.452},"max":{"1min":94.155000000000001,"5min":94.155000000000001,"15min":239.24600000000001},"last":13.516999999999999},{"interface":"front","average":{"1min":16.765000000000001,"5min":13.634,"15min":18.114999999999998},"min":{"1min":8.3179999999999996,"5min":6.8579999999999997,"15min":5.6520000000000001},"max":{"1min":86.099999999999994,"5min":86.099999999999994,"15min":246.28299999999999},"last":10.23}]},{"osd":3,"last update":"Fri Dec 1 17:36:25 2023","interfaces":[{"interface":"back","average":{"1min":20.655000000000001,"5min":12.295999999999999,"15min":19.577000000000002},"min":{"1min":4.7889999999999997,"5min":4.7889999999999997,"15min":4.7889999999999997},"max":{"1min":128.08099999999999,"5min":128.08099999999999,"15min":240.017},"last":7.5229999999999997},{"interface":"front","average":{"1min":28.396000000000001,"5min":17.071999999999999,"15min":25.015999999999998},"min":{"1min":9.452,"5min":8.3499999999999996,"15min":7.2779999999999996},"max":{"1min":136.40600000000001,"5min":136.40600000000001,"15min":249.86099999999999},"last":14.917}]},{"osd":4,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":23.562999999999999,"5min":14.747999999999999,"15min":19.128},"min":{"1min":7.5369999999999999,"5min":6.6909999999999998,"15min":6.6909999999999998},"max":{"1min":196.71799999999999,"5min":196.71799999999999,"15min":241.292},"last":13.890000000000001},{"interface":"front","average":{"1min":27.734999999999999,"5min":19.274999999999999,"15min":22.911000000000001},"min":{"1min":9.625,"5min":9.3759999999999994,"15min":7.2290000000000001},"max":{"1min":207.38,"5min":207.38,"15min":236.65799999999999},"last":19.213000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":18.257000000000001,"5min":15.630000000000001,"15min":23.594999999999999},"min":{"1min":10.617000000000001,"5min":8.6929999999999996,"15min":7.9400000000000004},"max":{"1min":55.027000000000001,"5min":55.027000000000001,"15min":241.91200000000001},"last":17.364999999999998},{"interface":"front","average":{"1min":17.329999999999998,"5min":14.760999999999999,"15min":20.811},"min":{"1min":8.3170000000000002,"5min":7.2779999999999996,"15min":6.4279999999999999},"max":{"1min":73.453000000000003,"5min":73.453000000000003,"15min":247.185},"last":10.845000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":20.111000000000001,"5min":16.745999999999999,"15min":23.600000000000001},"min":{"1min":10.632,"5min":8.2829999999999995,"15min":7.923},"max":{"1min":68.665999999999997,"5min":68.665999999999997,"15min":244.93799999999999},"last":18.041},{"interface":"front","average":{"1min":19.872,"5min":15.91,"15min":22.402999999999999},"min":{"1min":11.055999999999999,"5min":9.2110000000000003,"15min":7.734},"max":{"1min":83.817999999999998,"5min":83.817999999999998,"15min":247.80500000000001},"last":14.188000000000001}]},{"osd":7,"last update":"Fri Dec 1 17:37:10 2023","interfaces":[{"interface":"back","average":{"1min":28.699000000000002,"5min":19.442,"15min":24.628},"min":{"1min":13.359,"5min":10.281000000000001,"15min":9.0820000000000007},"max":{"1min":199.01400000000001,"5min":199.01400000000001,"15min":238.58099999999999},"last":15.555999999999999},{"interface":"front","average":{"1min":20.789000000000001,"5min":17.556999999999999,"15min":24.472000000000001},"min":{"1min":11.659000000000001,"5min":9.3140000000000001,"15min":9.3140000000000001},"max":{"1min":91.361000000000004,"5min":91.361000000000004,"15min":273.142},"last":11.487}]}]},{"osd":6,"up_from":14,"seq":60129542544,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69604,"kb_used_data":6492,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302236,"statfs":{"total":96636764160,"available":96565489664,"internally_reserved":0,"allocated":6647808,"data_stored":3599223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":10.34,"5min":9.7390000000000008,"15min":13.387},"min":{"1min":4.8339999999999996,"5min":4.577,"15min":4.0730000000000004},"max":{"1min":25.731999999999999,"5min":25.731999999999999,"15min":129.89699999999999},"last":7.7709999999999999},{"interface":"front","average":{"1min":11.954000000000001,"5min":10.832000000000001,"15min":15.518000000000001},"min":{"1min":6.5010000000000003,"5min":6.4669999999999996,"15min":4.8129999999999997},"max":{"1min":34.526000000000003,"5min":34.526000000000003,"15min":155.99100000000001},"last":11.359999999999999}]},{"osd":1,"last update":"Fri Dec 1 17:37:02 2023","interfaces":[{"interface":"back","average":{"1min":13.353999999999999,"5min":12.460000000000001,"15min":15.26},"min":{"1min":6.9640000000000004,"5min":6.9450000000000003,"15min":6.2789999999999999},"max":{"1min":43.774000000000001,"5min":50.130000000000003,"15min":175.55000000000001},"last":10.624000000000001},{"interface":"front","average":{"1min":14.955,"5min":15.695,"15min":19.050999999999998},"min":{"1min":9.7680000000000007,"5min":9.7680000000000007,"15min":7.5369999999999999},"max":{"1min":38.823,"5min":54.060000000000002,"15min":130.91999999999999},"last":16.805}]},{"osd":2,"last update":"Fri Dec 1 17:36:11 2023","interfaces":[{"interface":"back","average":{"1min":21.776,"5min":15.327,"15min":19.893999999999998},"min":{"1min":7.0010000000000003,"5min":6.8879999999999999,"15min":6.8879999999999999},"max":{"1min":172.459,"5min":172.459,"15min":191.71299999999999},"last":11.997999999999999},{"interface":"front","average":{"1min":19.948,"5min":11.276999999999999,"15min":16.713000000000001},"min":{"1min":4.4370000000000003,"5min":4.4370000000000003,"15min":4.4370000000000003},"max":{"1min":174.608,"5min":174.608,"15min":244.46299999999999},"last":17.489000000000001}]},{"osd":3,"last update":"Fri Dec 1 17:37:02 2023","interfaces":[{"interface":"back","average":{"1min":16.753,"5min":15.779,"15min":20.751000000000001},"min":{"1min":11.468999999999999,"5min":9.4849999999999994,"15min":6.5830000000000002},"max":{"1min":32.539000000000001,"5min":70.066999999999993,"15min":223.97800000000001},"last":13.957000000000001},{"interface":"front","average":{"1min":16.844000000000001,"5min":15.34,"15min":22.016999999999999},"min":{"1min":9.4120000000000008,"5min":7.2080000000000002,"15min":6.8559999999999999},"max":{"1min":65.215999999999994,"5min":71.659999999999997,"15min":212.309},"last":7.0899999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:51 2023","interfaces":[{"interface":"back","average":{"1min":22.206,"5min":16.388000000000002,"15min":19.161000000000001},"min":{"1min":9.109,"5min":8.9860000000000007,"15min":7.0510000000000002},"max":{"1min":67.287000000000006,"5min":67.287000000000006,"15min":134.95699999999999},"last":9.9789999999999992},{"interface":"front","average":{"1min":25.456,"5min":19.428999999999998,"15min":22.629999999999999},"min":{"1min":13.336,"5min":11.27,"15min":9.9809999999999999},"max":{"1min":74.200999999999993,"5min":74.200999999999993,"15min":135.798},"last":12.593}]},{"osd":5,"last update":"Fri Dec 1 17:36:51 2023","interfaces":[{"interface":"back","average":{"1min":21.643000000000001,"5min":16.739000000000001,"15min":21.106000000000002},"min":{"1min":7.9909999999999997,"5min":7.8040000000000003,"15min":7.2869999999999999},"max":{"1min":70.823999999999998,"5min":70.823999999999998,"15min":288.577},"last":18.317},{"interface":"front","average":{"1min":26.931000000000001,"5min":19.710000000000001,"15min":23.295999999999999},"min":{"1min":12.140000000000001,"5min":12.140000000000001,"15min":10.234},"max":{"1min":96.646000000000001,"5min":96.646000000000001,"15min":142.77699999999999},"last":13.314}]},{"osd":7,"last update":"Fri Dec 1 17:36:51 2023","interfaces":[{"interface":"back","average":{"1min":16.306999999999999,"5min":11.847,"15min":15.145},"min":{"1min":6.5309999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"max":{"1min":59.734000000000002,"5min":59.734000000000002,"15min":168.59700000000001},"last":9.0779999999999994},{"interface":"front","average":{"1min":23.439,"5min":18.123000000000001,"15min":21.085000000000001},"min":{"1min":14.02,"5min":8.9459999999999997,"15min":8.4770000000000003},"max":{"1min":69.829999999999998,"5min":69.829999999999998,"15min":183.054},"last":14.618}]}]},{"osd":5,"up_from":14,"seq":60129542545,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74332,"kb_used_data":7060,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297508,"statfs":{"total":96636764160,"available":96560648192,"internally_reserved":0,"allocated":7229440,"data_stored":4115472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:37:16 2023","interfaces":[{"interface":"back","average":{"1min":11.666,"5min":12.15,"15min":15.135},"min":{"1min":4.6619999999999999,"5min":4.1360000000000001,"15min":4.1360000000000001},"max":{"1min":73.542000000000002,"5min":201.333,"15min":228.261},"last":6.4390000000000001},{"interface":"front","average":{"1min":15.208,"5min":14.106999999999999,"15min":17.170000000000002},"min":{"1min":6.3040000000000003,"5min":5.0339999999999998,"15min":4.6580000000000004},"max":{"1min":85.289000000000001,"5min":221.876,"15min":222.441},"last":7.056}]},{"osd":1,"last update":"Fri Dec 1 17:37:07 2023","interfaces":[{"interface":"back","average":{"1min":16.920999999999999,"5min":13.304,"15min":19.277000000000001},"min":{"1min":6.3460000000000001,"5min":6.0640000000000001,"15min":4.8899999999999997},"max":{"1min":72.462999999999994,"5min":72.462999999999994,"15min":216.20099999999999},"last":9.2579999999999991},{"interface":"front","average":{"1min":15.742000000000001,"5min":13.532,"15min":20.065999999999999},"min":{"1min":7.3710000000000004,"5min":6.5650000000000004,"15min":6.2539999999999996},"max":{"1min":31.949999999999999,"5min":65.156999999999996,"15min":255.95599999999999},"last":8.4809999999999999}]},{"osd":2,"last update":"Fri Dec 1 17:37:01 2023","interfaces":[{"interface":"back","average":{"1min":29.966000000000001,"5min":15.294,"15min":18.52},"min":{"1min":7.2530000000000001,"5min":6.468,"15min":6.468},"max":{"1min":226.881,"5min":226.881,"15min":226.881},"last":10.169},{"interface":"front","average":{"1min":31.094999999999999,"5min":18.686,"15min":23.234000000000002},"min":{"1min":10.438000000000001,"5min":8.7070000000000007,"15min":8.7070000000000007},"max":{"1min":200.06800000000001,"5min":200.06800000000001,"15min":224.47300000000001},"last":12.332000000000001}]},{"osd":3,"last update":"Fri Dec 1 17:37:07 2023","interfaces":[{"interface":"back","average":{"1min":17.044,"5min":13.977,"15min":18.629000000000001},"min":{"1min":7.1609999999999996,"5min":6.819,"15min":5.9020000000000001},"max":{"1min":95.180000000000007,"5min":95.180000000000007,"15min":248.07300000000001},"last":14.917999999999999},{"interface":"front","average":{"1min":20.901,"5min":17.681000000000001,"15min":22.684999999999999},"min":{"1min":8.9320000000000004,"5min":8.9320000000000004,"15min":8.9320000000000004},"max":{"1min":85.072000000000003,"5min":85.072000000000003,"15min":235.02699999999999},"last":17.292999999999999}]},{"osd":4,"last update":"Fri Dec 1 17:36:55 2023","interfaces":[{"interface":"back","average":{"1min":19.247,"5min":16.109000000000002,"15min":19.242000000000001},"min":{"1min":8.3200000000000003,"5min":8.3200000000000003,"15min":8.3200000000000003},"max":{"1min":36.350999999999999,"5min":59.747,"15min":211.09399999999999},"last":14.119999999999999},{"interface":"front","average":{"1min":22.109999999999999,"5min":15.996,"15min":18.448},"min":{"1min":9.2189999999999994,"5min":8.6850000000000005,"15min":7.0540000000000003},"max":{"1min":114.05200000000001,"5min":114.05200000000001,"15min":216.85300000000001},"last":10.733000000000001}]},{"osd":6,"last update":"Fri Dec 1 17:36:55 2023","interfaces":[{"interface":"back","average":{"1min":22.838999999999999,"5min":18.434999999999999,"15min":21.120000000000001},"min":{"1min":10.314,"5min":10.103,"15min":8.4339999999999993},"max":{"1min":81.772999999999996,"5min":111.396,"15min":223.89500000000001},"last":15.750999999999999},{"interface":"front","average":{"1min":19.198,"5min":16.227,"15min":14.467000000000001},"min":{"1min":8.0660000000000007,"5min":7.0220000000000002,"15min":5.6479999999999997},"max":{"1min":59.250999999999998,"5min":59.250999999999998,"15min":159.84399999999999},"last":11.731999999999999}]},{"osd":7,"last update":"Fri Dec 1 17:36:55 2023","interfaces":[{"interface":"back","average":{"1min":23.693999999999999,"5min":18.780999999999999,"15min":21.957000000000001},"min":{"1min":10.323,"5min":10.323,"15min":9.6430000000000007},"max":{"1min":93.954999999999998,"5min":93.954999999999998,"15min":226.06100000000001},"last":12.932},{"interface":"front","average":{"1min":21.358000000000001,"5min":18.934000000000001,"15min":21.984999999999999},"min":{"1min":8.9890000000000008,"5min":8.9890000000000008,"15min":8.9559999999999995},"max":{"1min":49.155000000000001,"5min":125.431,"15min":317.267},"last":16.513000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":74132,"kb_used_data":6860,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94297708,"statfs":{"total":96636764160,"available":96560852992,"internally_reserved":0,"allocated":7024640,"data_stored":3846006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:37:11 2023","interfaces":[{"interface":"back","average":{"1min":10.666,"5min":9.5869999999999997,"15min":15.901999999999999},"min":{"1min":4.0300000000000002,"5min":4.0300000000000002,"15min":3.7690000000000001},"max":{"1min":42.569000000000003,"5min":42.569000000000003,"15min":521.55899999999997},"last":7.6980000000000004},{"interface":"front","average":{"1min":11.318,"5min":10.090999999999999,"15min":14.997999999999999},"min":{"1min":6.3499999999999996,"5min":5.3470000000000004,"15min":5.3470000000000004},"max":{"1min":37.837000000000003,"5min":37.837000000000003,"15min":281.125},"last":10.432}]},{"osd":1,"last update":"Fri Dec 1 17:37:11 2023","interfaces":[{"interface":"back","average":{"1min":13.493,"5min":11.763999999999999,"15min":18.568000000000001},"min":{"1min":7.0439999999999996,"5min":6.7969999999999997,"15min":6.6050000000000004},"max":{"1min":66.022999999999996,"5min":66.022999999999996,"15min":308.99299999999999},"last":15.644},{"interface":"front","average":{"1min":20.515000000000001,"5min":16.670999999999999,"15min":22.881},"min":{"1min":11.068,"5min":10.298,"15min":6.3079999999999998},"max":{"1min":97.090000000000003,"5min":97.090000000000003,"15min":289.79399999999998},"last":18.957000000000001}]},{"osd":2,"last update":"Fri Dec 1 17:37:05 2023","interfaces":[{"interface":"back","average":{"1min":19.914000000000001,"5min":15.911,"15min":21.635000000000002},"min":{"1min":10.244,"5min":9.1029999999999998,"15min":5.976},"max":{"1min":80.295000000000002,"5min":80.295000000000002,"15min":298.18200000000002},"last":17.823},{"interface":"front","average":{"1min":12.634,"5min":11.204000000000001,"15min":15.894},"min":{"1min":6.0659999999999998,"5min":6.0659999999999998,"15min":6.0659999999999998},"max":{"1min":38.761000000000003,"5min":38.761000000000003,"15min":223.768},"last":7.2359999999999998}]},{"osd":3,"last update":"Fri Dec 1 17:37:11 2023","interfaces":[{"interface":"back","average":{"1min":17.062000000000001,"5min":13.456,"15min":19.059000000000001},"min":{"1min":8.0009999999999994,"5min":7.1580000000000004,"15min":6.2169999999999996},"max":{"1min":100.527,"5min":100.527,"15min":243.40299999999999},"last":9.3989999999999991},{"interface":"front","average":{"1min":19.545000000000002,"5min":17.042999999999999,"15min":22.306999999999999},"min":{"1min":12.026999999999999,"5min":10.156000000000001,"15min":8.4390000000000001},"max":{"1min":99.075999999999993,"5min":99.075999999999993,"15min":253.56100000000001},"last":12.92}]},{"osd":5,"last update":"Fri Dec 1 17:37:05 2023","interfaces":[{"interface":"back","average":{"1min":22.849,"5min":17.126000000000001,"15min":23.295000000000002},"min":{"1min":8.7010000000000005,"5min":8.7010000000000005,"15min":7.7290000000000001},"max":{"1min":135.33500000000001,"5min":135.33500000000001,"15min":336.55599999999998},"last":12.196},{"interface":"front","average":{"1min":19.417999999999999,"5min":12.583,"15min":19.271999999999998},"min":{"1min":6.3200000000000003,"5min":6.3200000000000003,"15min":6.3200000000000003},"max":{"1min":138.595,"5min":138.595,"15min":357.50900000000001},"last":14.859}]},{"osd":6,"last update":"Fri Dec 1 17:37:05 2023","interfaces":[{"interface":"back","average":{"1min":18.943999999999999,"5min":16.234999999999999,"15min":20.710999999999999},"min":{"1min":8.2089999999999996,"5min":8.2089999999999996,"15min":8.2089999999999996},"max":{"1min":98.460999999999999,"5min":98.460999999999999,"15min":255.52500000000001},"last":17.010000000000002},{"interface":"front","average":{"1min":22.34,"5min":16.704000000000001,"15min":19.420999999999999},"min":{"1min":9.2769999999999992,"5min":7.7080000000000002,"15min":7.7080000000000002},"max":{"1min":101.78400000000001,"5min":101.78400000000001,"15min":312.791},"last":14.041}]},{"osd":7,"last update":"Fri Dec 1 17:37:05 2023","interfaces":[{"interface":"back","average":{"1min":17.48,"5min":16.068000000000001,"15min":22.739999999999998},"min":{"1min":11.038,"5min":9.7590000000000003,"15min":8.7330000000000005},"max":{"1min":44.503,"5min":44.503,"15min":413.48899999999998},"last":19.829999999999998},{"interface":"front","average":{"1min":17.733000000000001,"5min":16.646999999999998,"15min":21.291},"min":{"1min":8.6769999999999996,"5min":8.6769999999999996,"15min":6.6820000000000004},"max":{"1min":41.779000000000003,"5min":41.779000000000003,"15min":417.952},"last":20.57}]}]},{"osd":7,"up_from":13,"seq":55834575251,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69780,"kb_used_data":6668,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94302060,"statfs":{"total":96636764160,"available":96565309440,"internally_reserved":0,"allocated":6828032,"data_stored":3717632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":64616906},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":13.364000000000001,"5min":11.880000000000001,"15min":16.117000000000001},"min":{"1min":8.8580000000000005,"5min":5.2729999999999997,"15min":4.3360000000000003},"max":{"1min":39.423000000000002,"5min":39.423000000000002,"15min":203.19300000000001},"last":15.632},{"interface":"front","average":{"1min":17.373999999999999,"5min":15.071999999999999,"15min":18.241},"min":{"1min":11.513,"5min":6.4269999999999996,"15min":4.4199999999999999},"max":{"1min":40.698999999999998,"5min":40.698999999999998,"15min":204.13},"last":16.821000000000002}]},{"osd":1,"last update":"Fri Dec 1 17:36:58 2023","interfaces":[{"interface":"back","average":{"1min":23.890000000000001,"5min":11.366,"15min":15.542},"min":{"1min":6.016,"5min":5.2300000000000004,"15min":5.0369999999999999},"max":{"1min":217.70500000000001,"5min":217.70500000000001,"15min":217.70500000000001},"last":9.8379999999999992},{"interface":"front","average":{"1min":30.745999999999999,"5min":17.751000000000001,"15min":17.585000000000001},"min":{"1min":12.067,"5min":7.4870000000000001,"15min":5.484},"max":{"1min":216.892,"5min":216.892,"15min":216.892},"last":20.617999999999999}]},{"osd":2,"last update":"Fri Dec 1 17:36:23 2023","interfaces":[{"interface":"back","average":{"1min":11.753,"5min":10.013999999999999,"15min":16.608000000000001},"min":{"1min":5.8700000000000001,"5min":5.1820000000000004,"15min":5.1820000000000004},"max":{"1min":26.274999999999999,"5min":26.274999999999999,"15min":230.40600000000001},"last":11.247999999999999},{"interface":"front","average":{"1min":14.275,"5min":12.394,"15min":17.012},"min":{"1min":5.5739999999999998,"5min":5.5739999999999998,"15min":5.5739999999999998},"max":{"1min":29.861000000000001,"5min":29.861000000000001,"15min":268.68000000000001},"last":10.882999999999999}]},{"osd":3,"last update":"Fri Dec 1 17:36:14 2023","interfaces":[{"interface":"back","average":{"1min":18.196000000000002,"5min":13.083,"15min":20.297000000000001},"min":{"1min":8.1359999999999992,"5min":6.5110000000000001,"15min":6.5110000000000001},"max":{"1min":85.724000000000004,"5min":85.724000000000004,"15min":290.15300000000002},"last":14.127000000000001},{"interface":"front","average":{"1min":14.601000000000001,"5min":11.683,"15min":20.149000000000001},"min":{"1min":6.6319999999999997,"5min":6.1079999999999997,"15min":6.1079999999999997},"max":{"1min":83.183000000000007,"5min":83.183000000000007,"15min":305.41399999999999},"last":11.986000000000001}]},{"osd":4,"last update":"Fri Dec 1 17:36:40 2023","interfaces":[{"interface":"back","average":{"1min":22.085999999999999,"5min":15.728999999999999,"15min":18.579000000000001},"min":{"1min":10.250999999999999,"5min":8.5489999999999995,"15min":6.9299999999999997},"max":{"1min":108.761,"5min":108.761,"15min":149.93000000000001},"last":18.021000000000001},{"interface":"front","average":{"1min":25.981999999999999,"5min":19.062999999999999,"15min":23.542000000000002},"min":{"1min":13.847,"5min":11.496,"15min":9.7810000000000006},"max":{"1min":109.95399999999999,"5min":109.95399999999999,"15min":239.37},"last":21.818000000000001}]},{"osd":5,"last update":"Fri Dec 1 17:36:40 2023","interfaces":[{"interface":"back","average":{"1min":30.864000000000001,"5min":17.353999999999999,"15min":21.384},"min":{"1min":11.603999999999999,"5min":7.3879999999999999,"15min":6.8460000000000001},"max":{"1min":124.021,"5min":124.021,"15min":282.31200000000001},"last":19.428000000000001},{"interface":"front","average":{"1min":29.943999999999999,"5min":16.628,"15min":20.617999999999999},"min":{"1min":7.1900000000000004,"5min":6.1529999999999996,"15min":6.1529999999999996},"max":{"1min":133.404,"5min":133.404,"15min":193.83600000000001},"last":12.706}]},{"osd":6,"last update":"Fri Dec 1 17:36:40 2023","interfaces":[{"interface":"back","average":{"1min":26.922000000000001,"5min":19.552,"15min":20.542000000000002},"min":{"1min":13.866,"5min":11.816000000000001,"15min":6.625},"max":{"1min":90.069000000000003,"5min":90.069000000000003,"15min":209.25},"last":23.013000000000002},{"interface":"front","average":{"1min":35.006,"5min":22.48,"15min":25.449999999999999},"min":{"1min":16.302,"5min":14.266999999999999,"15min":8.9420000000000002},"max":{"1min":150.14400000000001,"5min":150.14400000000001,"15min":193.184},"last":13.340999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-01T17:37:19.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.840+0000 7f8bb3bf0640 10 monclient: get_monmap_and_config 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.840+0000 7f8bb3bf0640 10 monclient: build_initial_monmap 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.840+0000 7f8bb3bf0640 10 monclient: monmap: 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:37:19.840883+0000 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:37:19.840883+0000 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:37:19.840 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.841 INFO:teuthology.orchestra.run.smithi067.stderr:0: v1:172.21.15.67:6789/0 mon.noname-a-legacy 2023-12-01T17:37:19.841 INFO:teuthology.orchestra.run.smithi067.stderr:1: v1:172.21.15.92:6789/0 mon.noname-c-legacy 2023-12-01T17:37:19.841 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-b 2023-12-01T17:37:19.841 INFO:teuthology.orchestra.run.smithi067.stderr:3: v2:172.21.15.67:3300/0 mon.noname-a 2023-12-01T17:37:19.841 INFO:teuthology.orchestra.run.smithi067.stderr:4: v2:172.21.15.92:3300/0 mon.noname-c 2023-12-01T17:37:19.841 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.840+0000 7f8bb3bf0640 1 Processor -- start 2023-12-01T17:37:19.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.840+0000 7f8bb3bf0640 1 -- start start 2023-12-01T17:37:19.843 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.840+0000 7f8bb3bf0640 10 monclient: init 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient: _reopen_session rank -1 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8bac0a8990 addr v1:172.21.15.67:6789/0 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8bac0a8d70 0x7f8bac1427b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient(hunting): picked mon.noname-a con 0x7f8bac0a8d70 addr v2:172.21.15.67:3300/0 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8bac13ef10 0x7f8bac13f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient(hunting): picked mon.noname-c con 0x7f8bac13ef10 addr v2:172.21.15.92:3300/0 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bac142cf0 con 0x7f8bac0a8990 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0aabe0 con 0x7f8bac0a8d70 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.844 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bac0aaec0 con 0x7f8bac13ef10 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient(hunting): _renew_subs 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.841+0000 7f8bb3bf0640 10 monclient(hunting): authenticate will time out at 3371.079049s 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb118b640 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f8bac0a8d70 0x7f8bac1427b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb218d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8bac13ef10 0x7f8bac13f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb218d640 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f8bac13ef10 0x7f8bac13f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:49090/0 (socket says 172.21.15.67:49090) 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb218d640 1 -- 172.21.15.67:0/2669332407 learned_addr learned my addr 172.21.15.67:0/2669332407 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb218d640 10 monclient(hunting): get_auth_request con 0x7f8bac13ef10 auth_method 0 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb218d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb218d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb218d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb118b640 10 monclient(hunting): get_auth_request con 0x7f8bac0a8d70 auth_method 0 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb118b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb118b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.842+0000 7f8bb118b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.845 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.843+0000 7f8bb218d640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.846 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.843+0000 7f8bb218d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.846 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.843+0000 7f8bb218d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.846 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.843+0000 7f8bb118b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.846 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.843+0000 7f8bb118b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.846 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.843+0000 7f8bb118b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 10 monclient(hunting): handle_auth_done global_id 23266 payload 274 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 1 -- 172.21.15.67:0/2669332407 >> v2:172.21.15.67:3300/0 conn(0x7f8bac0a8d70 msgr2=0x7f8bac1427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 1 --2- 172.21.15.67:0/2669332407 >> v2:172.21.15.67:3300/0 conn(0x7f8bac0a8d70 0x7f8bac1427b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 1 -- 172.21.15.67:0/2669332407 >> v1:172.21.15.67:6789/0 conn(0x7f8bac0a8990 legacy=0x7f8bac0b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 10 monclient: _finish_hunting 0 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 1 monclient: found mon.noname-c 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.847 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.844+0000 7f8bb218d640 1 -- 172.21.15.67:0/2669332407 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac142260 con 0x7f8bac13ef10 2023-12-01T17:37:19.848 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.846+0000 7f8bb218d640 1 --2- 172.21.15.67:0/2669332407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac13ef10 0x7f8bac13f380 secure :-1 s=READY pgs=5920 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0b3d20 tx=0x7f8ba402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e540125c0df27bf server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:37:19.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 1 -- 172.21.15.67:0/2669332407 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba402cb20 con 0x7f8bac13ef10 2023-12-01T17:37:19.850 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: dump: 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: _finish_auth 0 2023-12-01T17:37:19.851 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: _check_auth_tickets 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb3bf0640 5 monclient: authenticate success, global_id 23266 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 1 -- 172.21.15.67:0/2669332407 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba402cc80 con 0x7f8bac13ef10 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb3bf0640 10 monclient: get_monmap_and_config success 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb3bf0640 4 set_mon_vals no callback set 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 1 -- 172.21.15.67:0/2669332407 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba40343f0 con 0x7f8bac13ef10 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:37:19.852 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.850+0000 7f8bb098a640 10 monclient: dump: 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 10 monclient: shutdown 2023-12-01T17:37:19.853 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2669332407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac13ef10 msgr2=0x7f8bac13f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2669332407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac13ef10 0x7f8bac13f380 secure :-1 s=READY pgs=5920 cs=0 l=1 rev1=1 crypto rx=0x7f8bac0b3d20 tx=0x7f8ba402bf80 comp rx=0 tx=0).stop 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2669332407 shutdown_connections 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2669332407 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac13ef10 0x7f8bac13f380 unknown :-1 s=CLOSED pgs=5920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2669332407 >> v2:172.21.15.67:3300/0 conn(0x7f8bac0a8d70 0x7f8bac1427b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2669332407 >> 172.21.15.67:0/2669332407 conn(0x7f8bac0b8310 msgr2=0x7f8bac0b42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2669332407 shutdown_connections 2023-12-01T17:37:19.854 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2669332407 wait complete. 2023-12-01T17:37:19.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 10 monclient: build_initial_monmap 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.851+0000 7f8bb3bf0640 10 monclient: monmap: 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 0 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 0.000000 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:created 0.000000 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 0 (unknown) 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.noname-a 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.noname-b 2023-12-01T17:37:19.859 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.noname-c 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 librados: starting msgr at 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 librados: starting objecter 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 Processor -- start 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 -- start start 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 librados: setting wanted keys 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 librados: calling monclient init 2023-12-01T17:37:19.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient: init 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient: _reopen_session rank -1 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient: _add_conns ranks=[1,0,2] 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac0a8990 0x7f8bac13a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): picked mon.noname-b con 0x7f8bac0a8990 addr [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bac0a8d70 0x7f8bac139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): picked mon.noname-a con 0x7f8bac0a8d70 addr [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 --2- >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8bac13ef10 0x7f8bac137a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.861 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): picked mon.noname-c con 0x7f8bac13ef10 addr [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac13b040 con 0x7f8bac0a8d70 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac13a530 con 0x7f8bac0a8990 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): start opening mon connection 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 1 -- --> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac13b4d0 con 0x7f8bac13ef10 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): _renew_subs 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.852+0000 7f8bb3bf0640 10 monclient(hunting): authenticate will time out at 3371.090289s 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb198c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac0a8990 0x7f8bac13a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb118b640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bac0a8d70 0x7f8bac139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb198c640 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac0a8990 0x7f8bac13a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.67:49104/0 (socket says 172.21.15.67:49104) 2023-12-01T17:37:19.862 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb198c640 1 -- 172.21.15.67:0/2462372706 learned_addr learned my addr 172.21.15.67:0/2462372706 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb198c640 10 monclient(hunting): get_auth_request con 0x7f8bac0a8990 auth_method 0 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb198c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb198c640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb198c640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb118b640 10 monclient(hunting): get_auth_request con 0x7f8bac0a8d70 auth_method 0 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb118b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb118b640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.853+0000 7f8bb118b640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.854+0000 7f8bb198c640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.854+0000 7f8bb198c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.854+0000 7f8bb198c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.854+0000 7f8bb118b640 10 monclient(hunting): handle_auth_reply_more payload 9 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.854+0000 7f8bb118b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-12-01T17:37:19.863 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.854+0000 7f8bb118b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.855+0000 7f8bb218d640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8bac13ef10 0x7f8bac137a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb218d640 10 monclient(hunting): get_auth_request con 0x7f8bac13ef10 auth_method 0 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb218d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb218d640 10 monclient(hunting): _init_auth method 2 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb218d640 10 monclient(hunting): _init_auth creating new auth 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 10 monclient(hunting): handle_auth_done global_id 23269 payload 824 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 1 -- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8bac13ef10 msgr2=0x7f8bac137a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8bac13ef10 0x7f8bac137a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 1 -- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bac0a8d70 msgr2=0x7f8bac139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bac0a8d70 0x7f8bac139ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 10 monclient: _finish_hunting 0 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 1 monclient: found mon.noname-b 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.856+0000 7f8bb198c640 1 -- 172.21.15.67:0/2462372706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba4009b30 con 0x7f8bac0a8990 2023-12-01T17:37:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.858+0000 7f8bb198c640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac0a8990 0x7f8bac13a9b0 secure :-1 s=READY pgs=5921 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800c400 tx=0x7f8ba800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:37:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.858+0000 7f8ba27fc640 10 client.?.objecter ms_handle_connect 0x7f8bac0a8990 2023-12-01T17:37:19.865 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.858+0000 7f8ba27fc640 10 client.?.objecter resend_mon_ops 2023-12-01T17:37:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba80052d0 con 0x7f8bac0a8990 2023-12-01T17:37:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-12-01T17:37:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: dump: 2023-12-01T17:37:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.866 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: _finish_auth 0 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: _check_auth_tickets 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2023-12-01T17:37:19.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 5 monclient: authenticate success, global_id 23269 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba8005450 con 0x7f8bac0a8990 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 10 monclient: _renew_subs 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac0abb80 con 0x7f8bac0a8990 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: handle_config config(0 keys) v1 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba8007260 con 0x7f8bac0a8990 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba27fc640 10 monclient: dump: 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:epoch 1 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:fsid 2c621237-d4b8-4504-931d-6a9429822e96 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:last_changed 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.868 INFO:teuthology.orchestra.run.smithi067.stderr:created 2023-12-01T17:02:12.960958+0000 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:min_mon_release 18 (reef) 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:election_strategy: 1 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:1: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.b 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2: [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] mon.c 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 10 client.23269.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 10 monclient: _renew_subs 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8ba3fff640 4 set_mon_vals no callback set 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac137f50 con 0x7f8bac0a8990 2023-12-01T17:37:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.862+0000 7f8bb3bf0640 1 librados: init done 2023-12-01T17:37:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.863+0000 7f8bb3bf0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-12-01T17:37:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.863+0000 7f8bb3bf0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:37:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.863+0000 7f8bb3bf0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:37:19.870 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.863+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac04e840 con 0x7f8bac0a8990 2023-12-01T17:37:19.879 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.880+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 89231+0+0 (secure 0 0 0) 0x7f8ba80073c0 con 0x7f8bac0a8990 2023-12-01T17:37:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.880+0000 7f8ba27fc640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c8a0 0x7f8b9006ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-12-01T17:37:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.881+0000 7f8bb118b640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c8a0 0x7f8b9006ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-12-01T17:37:19.880 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.881+0000 7f8bb118b640 10 monclient: get_auth_request con 0x7f8b9006c8a0 auth_method 0 2023-12-01T17:37:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.881+0000 7f8ba27fc640 10 client.23269.objecter ms_handle_connect 0x7f8b9006c8a0 2023-12-01T17:37:19.881 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.881+0000 7f8bb118b640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c8a0 0x7f8b9006ed60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8bac133d70 tx=0x7f8b9c009190 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-01T17:37:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.882+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 5 ==== osd_map(1099..1099 src has 1049..1099) v4 ==== 6393+0+0 (secure 0 0 0) 0x7f8ba8058040 con 0x7f8bac0a8990 2023-12-01T17:37:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.882+0000 7f8ba27fc640 10 client.23269.objecter ms_dispatch 0x7f8bac000b90 osd_map(1099..1099 src has 1049..1099) v4 2023-12-01T17:37:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.882+0000 7f8ba27fc640 3 client.23269.objecter handle_osd_map got epochs [1099,1099] > 0 2023-12-01T17:37:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.882+0000 7f8ba27fc640 3 client.23269.objecter handle_osd_map decoding full epoch 1099 2023-12-01T17:37:19.882 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:19.883+0000 7f8ba27fc640 20 client.23269.objecter dump_active .. 0 homeless 2023-12-01T17:37:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.120+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ba8054380 con 0x7f8bac0a8990 2023-12-01T17:37:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.120+0000 7f8ba27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-12-01T17:37:20.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.120+0000 7f8ba27fc640 10 monclient: _finish_command 1 = system:0 2023-12-01T17:37:20.262 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:34.043 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.266 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:34.048 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.282 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:34.064 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.304+0000 7f8bb3bf0640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-01T17:37:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.304+0000 7f8bb3bf0640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-01T17:37:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.304+0000 7f8bb3bf0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.92:3300/0 2023-12-01T17:37:20.303 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.304+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f8bac0681a0 con 0x7f8bac0a8990 2023-12-01T17:37:20.311 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:34.072 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.314 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:34.075 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.327 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:34.088 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.330 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:34.112 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.332 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:34.114 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.345 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:34.127 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:20.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.937+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8ba805c0e0 con 0x7f8bac0a8990 2023-12-01T17:37:20.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.937+0000 7f8ba27fc640 10 monclient: handle_config config(1 keys) v1 2023-12-01T17:37:20.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.937+0000 7f8ba3fff640 4 set_mon_vals no callback set 2023-12-01T17:37:20.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.937+0000 7f8ba3fff640 10 set_mon_vals mon_health_to_clog = false 2023-12-01T17:37:20.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.937+0000 7f8ba27fc640 1 -- 172.21.15.67:0/2462372706 <== mon.1 v2:172.21.15.92:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f8ba80200c0 con 0x7f8bac0a8990 2023-12-01T17:37:20.937 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.938+0000 7f8ba27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-12-01T17:37:20.938 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.938+0000 7f8ba27fc640 10 monclient: _finish_command 2 = system:0 2023-12-01T17:37:20.939 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 10 librados: watch_flush enter 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 10 librados: watch_flush exit 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 20 client.23269.objecter shutdown clearing up homeless session... 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 10 client.23269.objecter successfully canceled tick 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c8a0 msgr2=0x7f8b9006ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c8a0 0x7f8b9006ed60 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f8bac133d70 tx=0x7f8b9c009190 comp rx=0 tx=0).stop 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 10 monclient: shutdown 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 20 monclient: shutdown discarding 0 pending message(s) 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac0a8990 msgr2=0x7f8bac13a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-01T17:37:20.940 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac0a8990 0x7f8bac13a9b0 secure :-1 s=READY pgs=5921 cs=0 l=1 rev1=1 crypto rx=0x7f8ba800c400 tx=0x7f8ba800c8c0 comp rx=0 tx=0).stop 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 shutdown_connections 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:6800/1043126838,v1:172.21.15.67:6801/1043126838] conn(0x7f8b9006c8a0 0x7f8b9006ed60 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:3301/0,v1:172.21.15.67:6790/0] conn(0x7f8bac13ef10 0x7f8bac137a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8bac0a8d70 0x7f8bac139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 --2- 172.21.15.67:0/2462372706 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8bac0a8990 0x7f8bac13a9b0 unknown :-1 s=CLOSED pgs=5921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 >> 172.21.15.67:0/2462372706 conn(0x7f8bac0b8310 msgr2=0x7f8bac13bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 shutdown_connections 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 -- 172.21.15.67:0/2462372706 wait complete. 2023-12-01T17:37:20.941 INFO:teuthology.orchestra.run.smithi067.stderr:2023-12-01T17:37:20.940+0000 7f8bb3bf0640 1 librados: shutdown 2023-12-01T17:37:20.954 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-01T17:37:20.954 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-01T17:37:20.954 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-12-01T17:37:20.954 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:37:20.968 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:37:20.960+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 86867) UID: 0 2023-12-01T17:37:20.968 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-12-01T17:37:20.961+0000 b47e640 -1 osd.0 1099 *** Got signal Terminated *** 2023-12-01T17:37:35.256 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:49.038 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.305 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:49.066 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.306 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:49.088 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.308 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:49.069 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.326 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:49.086 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.331 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:49.113 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.358 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:49.139 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.363 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:49.145 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:35.379 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:49.161 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.270 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:54.051 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.318 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:54.100 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.332 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:54.114 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.365 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:54.125 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.380 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:54.140 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.387 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:54.169 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.390 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:54.172 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.391 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:54.151 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.405 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:54.187 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.971 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:54.731 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.972 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:54.732 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:40.983 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:34:54.744 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:41.114 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:54.896 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:41.115 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:54.897 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:41.128 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:54.909 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:41.129 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:54.910 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:41.131 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:34:54.913 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:41.141 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:34:54.923 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.353 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:09.135 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.357 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:09.139 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.372 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:09.154 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.401 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:09.161 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.403 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:09.164 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.404 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:09.186 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.408 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:09.190 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.416 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:09.177 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:37:55.428 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:09.210 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.405 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:29.166 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.406 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:29.167 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.408 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:29.190 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.411 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:29.193 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.426 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:29.186 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.435 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:29.217 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.460 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:29.242 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.463 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:29.245 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:15.492 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:29.273 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.423 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:34.204 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.426 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:34.208 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.439 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:34.221 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.473 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:34.234 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.474 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:34.235 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.484 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:34.245 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.499 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:34.280 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.501 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:34.282 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:20.515 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:34.297 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.460 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:49.241 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.463 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:49.244 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.484 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:49.266 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.513 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:49.274 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.516 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:49.277 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.520 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:49.302 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.522 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:49.304 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.530 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:49.291 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:35.538 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:49.320 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.477 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:54.258 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.479 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:54.261 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.488 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:35:54.270 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.526 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:54.308 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.528 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:54.310 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.529 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:54.289 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.530 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:54.291 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.539 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:35:54.321 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:40.541 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:35:54.302 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.547 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:09.307 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.548 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:09.308 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.553 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:09.335 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.555 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:09.337 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.568 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:09.329 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.577 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:09.359 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.607 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:09.388 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.608 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:09.390 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:38:55.626 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:09.408 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.577 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:14.359 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.580 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:14.362 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.592 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:14.374 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.625 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:14.385 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.627 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:14.387 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.637 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:14.398 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.644 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:14.426 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.650 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:14.431 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:00.662 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:14.444 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.634 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:29.415 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.639 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:29.421 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.658 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:29.439 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.693 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:29.453 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.695 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:29.456 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.706 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:29.488 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.708 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:29.490 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.713 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:29.473 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:15.726 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:29.508 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:16.881 INFO:tasks.ceph.osd.0:Stopped 2023-12-01T17:39:16.882 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-12-01T17:39:16.882 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:16.895 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:39:16.887+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 86871) UID: 0 2023-12-01T17:39:16.895 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-12-01T17:39:16.888+0000 b47e640 -1 osd.1 1107 *** Got signal Terminated *** 2023-12-01T17:39:20.672 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:34.453 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.673 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:34.455 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.684 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:34.466 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.713 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:34.474 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.716 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:34.477 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.723 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:34.504 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.724 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:34.506 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.728 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:34.488 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:20.736 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:34.517 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:22.222 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-12-01T17:39:22.984 INFO:tasks.ceph.osd.1:Stopped 2023-12-01T17:39:22.984 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-12-01T17:39:22.984 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:22.997 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:39:22.989+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 86882) UID: 0 2023-12-01T17:39:22.997 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-12-01T17:39:22.990+0000 b47e640 -1 osd.2 1109 *** Got signal Terminated *** 2023-12-01T17:39:28.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-12-01T17:39:28.232 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-12-01T17:39:29.085 INFO:tasks.ceph.osd.2:Stopped 2023-12-01T17:39:29.126 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-12-01T17:39:29.126 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:29.140 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:39:29.131+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 86884) UID: 0 2023-12-01T17:39:29.140 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-12-01T17:39:29.132+0000 b47e640 -1 osd.3 1111 *** Got signal Terminated *** 2023-12-01T17:39:34.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-12-01T17:39:34.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-12-01T17:39:35.227 INFO:tasks.ceph.osd.3:Stopped 2023-12-01T17:39:35.228 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-12-01T17:39:35.228 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:35.245 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:39:35.234+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 87383) UID: 0 2023-12-01T17:39:35.245 INFO:tasks.ceph.osd.4.smithi092.stderr:2023-12-01T17:39:35.235+0000 b47e640 -1 osd.4 1114 *** Got signal Terminated *** 2023-12-01T17:39:39.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-12-01T17:39:39.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-12-01T17:39:40.855 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:54.637 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.902 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:54.662 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.902 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:54.663 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.906 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:54.687 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.906 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:54.688 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.911 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:54.693 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.925 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:36:54.686 86849== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.935 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:36:54.716 86583== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:40.955 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:36:54.737 86581== Warning: unimplemented fcntl command: 1036 2023-12-01T17:39:41.330 INFO:tasks.ceph.osd.4:Stopped 2023-12-01T17:39:41.330 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-12-01T17:39:41.330 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:41.343 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:39:41.335+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 87386) UID: 0 2023-12-01T17:39:41.343 INFO:tasks.ceph.osd.5.smithi092.stderr:2023-12-01T17:39:41.336+0000 b47e640 -1 osd.5 1117 *** Got signal Terminated *** 2023-12-01T17:39:45.755 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-12-01T17:39:47.431 INFO:tasks.ceph.osd.5:Stopped 2023-12-01T17:39:47.432 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-12-01T17:39:47.432 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:47.444 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:39:47.436+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 87388) UID: 0 2023-12-01T17:39:47.445 INFO:tasks.ceph.osd.6.smithi092.stderr:2023-12-01T17:39:47.437+0000 b47e640 -1 osd.6 1120 *** Got signal Terminated *** 2023-12-01T17:39:53.534 INFO:tasks.ceph.osd.6:Stopped 2023-12-01T17:39:53.534 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-12-01T17:39:53.534 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:53.547 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-12-01T17:39:53.538+0000 b47e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 87390) UID: 0 2023-12-01T17:39:53.550 INFO:tasks.ceph.osd.7.smithi092.stderr:2023-12-01T17:39:53.540+0000 b47e640 -1 osd.7 1120 *** Got signal Terminated *** 2023-12-01T17:39:56.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-12-01T17:39:59.636 INFO:tasks.ceph.osd.7:Stopped 2023-12-01T17:39:59.636 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-01T17:39:59.636 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-12-01T17:39:59.637 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:59.664 INFO:tasks.ceph.mgr.y:Stopped 2023-12-01T17:39:59.665 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-12-01T17:39:59.665 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:59.692 INFO:tasks.ceph.mgr.x:Stopped 2023-12-01T17:39:59.692 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-01T17:39:59.693 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-12-01T17:39:59.693 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:39:59.719 INFO:tasks.ceph.mon.a.smithi067.stderr:2023-12-01T17:39:59.709+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 86545) UID: 0 2023-12-01T17:39:59.733 INFO:tasks.ceph.mon.a.smithi067.stderr:2023-12-01T17:39:59.711+0000 9f77640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-01T17:40:00.342 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:37:14.123 86581== 2023-12-01T17:40:00.342 INFO:tasks.ceph.mon.a.smithi067.stderr:==00:00:37:14.123 86581== Exit program on first error (--exit-on-first-error=yes) 2023-12-01T17:40:00.365 INFO:tasks.ceph.mon.a.smithi067.stderr:daemon-helper: command failed with exit status 42 2023-12-01T17:40:02.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-12-01T17:40:02.169 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-12-01T17:40:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-01T17:40:05.794 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2023-12-01T17:40:05.795 INFO:tasks.ceph.mon.a:Stopped 2023-12-01T17:40:05.796 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-12-01T17:40:05.796 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:40:05.811 INFO:tasks.ceph.mon.c.smithi067.stderr:2023-12-01T17:40:05.801+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 86547) UID: 0 2023-12-01T17:40:05.812 INFO:tasks.ceph.mon.c.smithi067.stderr:2023-12-01T17:40:05.802+0000 9f77640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-12-01T17:40:06.430 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:37:20.212 86583== 2023-12-01T17:40:06.431 INFO:tasks.ceph.mon.c.smithi067.stderr:==00:00:37:20.212 86583== Exit program on first error (--exit-on-first-error=yes) 2023-12-01T17:40:06.450 INFO:tasks.ceph.mon.c.smithi067.stderr:daemon-helper: command failed with exit status 42 2023-12-01T17:40:07.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-12-01T17:40:07.271 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-12-01T17:40:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-01T17:40:11.897 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2023-12-01T17:40:11.898 INFO:tasks.ceph.mon.c:Stopped 2023-12-01T17:40:11.898 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-12-01T17:40:11.898 INFO:teuthology.orchestra.run:waiting for 300 2023-12-01T17:40:11.913 INFO:tasks.ceph.mon.b.smithi092.stderr:2023-12-01T17:40:11.903+0000 9f77640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 86845) UID: 0 2023-12-01T17:40:11.913 INFO:tasks.ceph.mon.b.smithi092.stderr:2023-12-01T17:40:11.905+0000 9f77640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-12-01T17:40:12.373 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-12-01T17:44:50.865 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:42:04.624 86849== 2023-12-01T17:44:50.865 INFO:tasks.ceph.mon.b.smithi092.stderr:==00:00:42:04.624 86849== Exit program on first error (--exit-on-first-error=yes) 2023-12-01T17:44:50.881 INFO:tasks.ceph.mon.b.smithi092.stderr:daemon-helper: command failed with exit status 42 2023-12-01T17:44:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 42 2023-12-01T17:44:52.572 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2023-12-01T17:44:52.573 INFO:tasks.ceph.mon.b:Stopped 2023-12-01T17:44:52.573 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-01T17:44:52.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-12-01T17:44:52.617 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:44:52.617 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-12-01T17:44:52.718 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:44:52.718 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-12-01T17:44:52.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:44:52.779 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-12-01T17:44:52.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:44:52.832 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-12-01T17:44:52.884 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:44:52.884 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-12-01T17:44:53.254 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:44:53.254 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-12-01T17:44:53.317 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:44:53.317 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-12-01T17:44:53.373 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:44:53.373 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-12-01T17:44:53.431 INFO:tasks.ceph:Archiving mon data... 2023-12-01T17:44:53.433 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/data/mon.a.tgz 2023-12-01T17:44:53.433 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2023-12-01T17:44:53.452 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.fKV0KwpNVd 2023-12-01T17:44:53.452 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.fKV0KwpNVd 2023-12-01T17:44:53.623 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.fKV0KwpNVd 2023-12-01T17:44:53.714 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.fKV0KwpNVd is 459KB 2023-12-01T17:44:53.812 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.fKV0KwpNVd 2023-12-01T17:44:53.827 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/data/mon.c.tgz 2023-12-01T17:44:53.827 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2023-12-01T17:44:53.883 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.yK6MBR7nJN 2023-12-01T17:44:53.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.yK6MBR7nJN 2023-12-01T17:44:54.034 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.yK6MBR7nJN 2023-12-01T17:44:54.124 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.yK6MBR7nJN is 462KB 2023-12-01T17:44:54.220 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.yK6MBR7nJN 2023-12-01T17:44:54.235 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/data/mon.b.tgz 2023-12-01T17:44:54.235 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2023-12-01T17:44:54.253 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.UHjhzZ8Ash 2023-12-01T17:44:54.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.UHjhzZ8Ash 2023-12-01T17:44:54.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.UHjhzZ8Ash 2023-12-01T17:44:54.513 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.UHjhzZ8Ash is 462KB 2023-12-01T17:44:54.609 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.UHjhzZ8Ash 2023-12-01T17:44:54.624 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-01T17:44:54.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-01T17:44:54.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-12-01T17:44:54.690 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-12-01T17:44:54.690 DEBUG:teuthology.orchestra.run.smithi067:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-01T17:44:54.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-12-01T17:44:54.743 INFO:teuthology.orchestra.run.smithi067.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2023-12-01T17:44:54.743 INFO:teuthology.orchestra.run.smithi067.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2023-12-01T17:44:54.743 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2023-12-01T17:44:54.743 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2023-12-01T17:44:54.743 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2023-12-01T17:44:54.743 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2023-12-01T17:44:54.771 INFO:teuthology.orchestra.run.smithi092.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2023-12-01T17:44:54.771 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2023-12-01T17:44:54.771 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2023-12-01T17:44:54.771 INFO:tasks.ceph:Archiving crash dumps... 2023-12-01T17:44:54.773 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/remote/smithi067/crash 2023-12-01T17:44:54.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-01T17:44:54.805 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/remote/smithi092/crash 2023-12-01T17:44:54.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-01T17:44:54.837 INFO:tasks.ceph:Compressing logs... 2023-12-01T17:44:54.837 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-01T17:44:54.850 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-01T17:44:54.881 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91581.log 2023-12-01T17:44:54.927 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-12-01T17:44:54.927 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2023-12-01T17:44:54.927 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2023-12-01T17:44:54.927 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2023-12-01T17:44:54.927 INFO:teuthology.orchestra.run.smithi092.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/osd.6.log: 63.9% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr: 74.7%/var/log/ceph/valgrind/osd.4.log: -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87242.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87042.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.87242.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.87042.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.87242.log.gz 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87082.log 2023-12-01T17:44:54.928 INFO:teuthology.orchestra.run.smithi092.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.87042.log.gz 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87122.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.87082.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.87122.log: gzip -5 --verbose 88.4% -- replaced with /var/log/ceph/ceph-client.admin.87082.log.gz 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr: -- /var/log/ceph/ceph-client.admin.87293.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87122.log.gz 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.admin.87293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87202.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.87293.log.gz 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87162.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.87202.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87202.log.gz 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.87162.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87162.log.gz 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87002.log 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-mon.b.log: 92.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-01T17:44:54.929 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/ceph-client.admin.87002.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.87002.log.gz 2023-12-01T17:44:54.930 INFO:teuthology.orchestra.run.smithi092.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-01T17:44:54.939 INFO:teuthology.orchestra.run.smithi092.stderr: 94.8% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160256.log 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91581.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160214.log 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160256.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91581.log.gz 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143062.log 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160256.log.gz 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159667.log 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.160214.log.gz 2023-12-01T17:44:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.143062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160384.log 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159962.log 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91899.log 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.159667.log.gz 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91159.log 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.160384.log.gz 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.143062.log.gz 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92751.log 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159962.log.gz 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90980.log 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91899.log.gz 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92751.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91159.log.gz -5 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.91302.log 2023-12-01T17:44:54.946 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86587.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92939.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.90980.log.gz 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160172.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91302.log.gz 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92939.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160172.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- 88.3% -- replaced with /var/log/ceph/ceph-client.admin.160172.log.gz 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr: /var/log/ceph/valgrind/osd.0.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.86587.log.gz/var/log/ceph/valgrind/osd.2.log: 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 63.9% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 64.0% --verbose -- -- replaced with /var/log/ceph/valgrind/osd.0.log.gz /var/log/ceph/valgrind/mon.a.log 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:44:54.947 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr: 63.9% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92875.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr: 74.7% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90967.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr: 64.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132671.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93066.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.132671.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90671.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90967.log.gz 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91727.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91726.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91727.log: /var/log/ceph/ceph-client.admin.90671.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.91727.log.gz 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.90671.log.gz 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91298.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92563.log 2023-12-01T17:44:54.948 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91298.log: 90.8%/var/log/ceph/ceph-client.admin.91726.log: -- replaced with /var/log/ceph/ceph-client.admin.132671.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/ceph-client.admin.91298.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.159793.log 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92563.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91726.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86889.log 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159793.log: gzip -5 --verbose 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92939.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr: -- /var/log/ceph/ceph-client.admin.157979.log 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159793.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:gzip/var/log/ceph/ceph-client.admin.86889.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.92785.log 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr: 91.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159488.log 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93066.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.157979.log: 87.2%/var/log/ceph/ceph-client.admin.92785.log: -- replaced with /var/log/ceph/ceph-client.admin.86889.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159709.log 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159583.log 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159709.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159488.log.gz 2023-12-01T17:44:54.949 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.92875.log.gz 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90978.log 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159709.log.gz 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92400.log 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143920.log 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.157979.log.gz 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.159583.log.gz 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92400.log: gzip -5 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90978.log.gz 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr: --verbose -- /var/log/ceph/ceph-client.admin.159836.log 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.143920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157980.log 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160088.log 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.157980.log: 93.5% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.159836.log.gz 2023-12-01T17:44:54.950 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92751.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93251.log 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90984.log 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93251.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160088.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr: 90.5% 90.9% -- replaced with /var/log/ceph/ceph-client.admin.92563.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.143920.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93360.log 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.88852.log 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.90984.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92440.log 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.88852.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.157980.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.88852.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92785.log.gz 2023-12-01T17:44:54.951 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93020.log 2023-12-01T17:44:54.952 INFO:teuthology.orchestra.run.smithi067.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92400.log.gz 2023-12-01T17:44:54.952 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90756.log 2023-12-01T17:44:54.952 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92440.log: /var/log/ceph/ceph-client.admin.93020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92226.log 2023-12-01T17:44:54.952 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143922.log 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92226.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.90756.log.gz 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90974.log 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.143922.log: 90.5% 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93020.log.gz 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.143922.log.gz 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134401.log 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92416.log 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.134401.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.90974.log.gz 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160466.log 2023-12-01T17:44:54.953 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92416.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.134401.log.gz 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92294.log 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160466.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.93360.log.gz 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.160466.log.gz 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90981.log 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93297.log 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90981.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.90981.log.gz 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90971.log 2023-12-01T17:44:54.954 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93297.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.92294.log.gz 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157978.log 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90971.log: 95.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93216.log 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.157978.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90971.log.gz 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92541.log 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93216.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.157978.log.gz 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87815.log 2023-12-01T17:44:54.955 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92541.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92541.log.gz 2023-12-01T17:44:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92662.log 2023-12-01T17:44:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87815.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87815.log.gz 2023-12-01T17:44:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143061.log 2023-12-01T17:44:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92662.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.93216.log.gz 2023-12-01T17:44:54.956 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92252.log 2023-12-01T17:44:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.143061.log: 94.6% -- replaced with /var/log/ceph/ceph-client.admin.93297.log.gz 2023-12-01T17:44:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160298.log 2023-12-01T17:44:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92252.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.92662.log.gz 2023-12-01T17:44:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-12-01T17:44:54.957 INFO:teuthology.orchestra.run.smithi067.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.143061.log.gz 2023-12-01T17:44:54.957 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90799.log 2023-12-01T17:44:54.958 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.1.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.160298.log.gz 2023-12-01T17:44:54.958 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91647.log 2023-12-01T17:44:54.958 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90799.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.92226.log.gz 2023-12-01T17:44:54.958 INFO:teuthology.orchestra.run.smithi067.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91724.log 2023-12-01T17:44:54.958 INFO:teuthology.orchestra.run.smithi067.stderr: -- replaced with /var/log/ceph/ceph-client.admin.90799.log.gz 2023-12-01T17:44:54.958 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160340.log 2023-12-01T17:44:54.958 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91724.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.91647.log.gz 2023-12-01T17:44:54.959 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.91724.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.92845.log 2023-12-01T17:44:54.959 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:44:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93288.log 2023-12-01T17:44:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92845.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160340.log.gz 2023-12-01T17:44:54.959 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160426.log 2023-12-01T17:44:54.975 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93288.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.92252.log.gz 2023-12-01T17:44:54.975 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153676.log 2023-12-01T17:44:54.976 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160426.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160426.log.gz 2023-12-01T17:44:54.976 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91357.log 2023-12-01T17:44:54.976 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.153676.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.92845.log.gz 2023-12-01T17:44:54.977 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90713.log 2023-12-01T17:44:54.977 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91357.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91357.log.gz 2023-12-01T17:44:54.977 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.153676.log.gz 2023-12-01T17:44:54.978 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92641.log 2023-12-01T17:44:54.978 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93023.log 2023-12-01T17:44:54.978 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92641.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90713.log.gz 2023-12-01T17:44:54.978 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143921.log 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93023.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.93023.log.gz 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153678.log 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.143921.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.143921.log.gz 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92954.log 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.153678.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.93251.log.gz 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132669.log 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92954.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.153678.log.gz 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.132669.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.92641.log.gz 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92615.log 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.3.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.92954.log.gz 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143063.log 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92615.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.132669.log.gz 2023-12-01T17:44:54.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93155.log 2023-12-01T17:44:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.143063.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.143063.log.gz 2023-12-01T17:44:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:gzip 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92615.log.gz 2023-12-01T17:44:54.980 INFO:teuthology.orchestra.run.smithi067.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92297.log 2023-12-01T17:44:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90549.log 2023-12-01T17:44:54.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92297.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.93288.log.gz 2023-12-01T17:44:54.980 INFO:teuthology.orchestra.run.smithi067.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.93155.log.gz 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134402.log 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93188.log 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.134402.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.90549.log.gz 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134402.log.gz 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91129.log 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93188.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80723.log 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91129.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.91129.log.gz 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153677.log 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.tmp-client.admin.80723.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80723.log.gz 2023-12-01T17:44:54.984 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91723.log 2023-12-01T17:44:54.985 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.153677.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.153677.log.gz 2023-12-01T17:44:54.986 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93067.log 2023-12-01T17:44:54.986 INFO:teuthology.orchestra.run.smithi067.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.93188.log.gz/var/log/ceph/ceph-client.admin.91723.log: 2023-12-01T17:44:54.988 INFO:teuthology.orchestra.run.smithi067.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.91723.log.gz 2023-12-01T17:44:54.988 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91445.log 2023-12-01T17:44:54.988 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93182.log 2023-12-01T17:44:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93067.log: /var/log/ceph/ceph-client.admin.91445.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.91445.log.gz 2023-12-01T17:44:54.989 INFO:teuthology.orchestra.run.smithi067.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.93067.log.gz 2023-12-01T17:44:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90509.log 2023-12-01T17:44:54.989 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92859.log 2023-12-01T17:44:54.990 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93182.log: /var/log/ceph/ceph-client.admin.90509.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.90509.log.gz 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93182.log.gz 2023-12-01T17:44:54.990 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:44:54.991 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91521.log 2023-12-01T17:44:54.991 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153675.log 2023-12-01T17:44:54.992 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92859.log: /var/log/ceph/ceph-client.admin.91521.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.91521.log.gz 2023-12-01T17:44:55.005 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-12-01T17:44:55.007 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.153675.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.153675.log.gz 2023-12-01T17:44:55.008 INFO:teuthology.orchestra.run.smithi067.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.92416.log.gz 2023-12-01T17:44:55.017 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92554.log 2023-12-01T17:44:55.017 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160046.log 2023-12-01T17:44:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92554.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.92554.log.gz 2023-12-01T17:44:55.027 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.92297.log.gz 2023-12-01T17:44:55.031 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91355.log 2023-12-01T17:44:55.032 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92269.log 2023-12-01T17:44:55.032 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91355.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.160046.log.gz 2023-12-01T17:44:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160004.log 2023-12-01T17:44:55.033 INFO:teuthology.orchestra.run.smithi067.stderr: 87.3%/var/log/ceph/ceph-client.admin.92269.log: -- replaced with /var/log/ceph/ceph-client.admin.91355.log.gz 2023-12-01T17:44:55.033 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93061.log 2023-12-01T17:44:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160004.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.160004.log.gz 2023-12-01T17:44:55.035 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92243.log 2023-12-01T17:44:55.039 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93061.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.93061.log.gz 2023-12-01T17:44:55.039 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-12-01T17:44:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92243.log: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.92269.log.gz 2023-12-01T17:44:55.043 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93283.log 2023-12-01T17:44:55.048 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.0.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.92859.log.gz 2023-12-01T17:44:55.056 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159625.log 2023-12-01T17:44:55.060 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93283.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.93283.log.gz 2023-12-01T17:44:55.069 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153674.log 2023-12-01T17:44:55.071 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159625.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.159625.log.gz 2023-12-01T17:44:55.083 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159878.log 2023-12-01T17:44:55.088 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.153674.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.153674.log.gz 2023-12-01T17:44:55.088 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-12-01T17:44:55.089 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159878.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.159878.log.gz 2023-12-01T17:44:55.090 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160130.log 2023-12-01T17:44:55.103 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93365.log 2023-12-01T17:44:55.104 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.160130.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160130.log.gz 2023-12-01T17:44:55.105 INFO:teuthology.orchestra.run.smithi067.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2023-12-01T17:44:55.106 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-12-01T17:44:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132668.log 2023-12-01T17:44:55.126 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-osd.2.log: 92.4% -- replaced with /var/log/ceph/ceph-client.admin.92243.log.gz 2023-12-01T17:44:55.133 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90619.log 2023-12-01T17:44:55.137 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.132668.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.132668.log.gz 2023-12-01T17:44:55.149 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159751.log 2023-12-01T17:44:55.152 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90619.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.90619.log.gz 2023-12-01T17:44:55.153 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159540.log 2023-12-01T17:44:55.154 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159751.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.159751.log.gz 2023-12-01T17:44:55.167 INFO:teuthology.orchestra.run.smithi067.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.93365.log.gz 2023-12-01T17:44:55.167 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92951.log 2023-12-01T17:44:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159540.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159540.log.gz 2023-12-01T17:44:55.168 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.86994.log 2023-12-01T17:44:55.169 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92951.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92951.log.gz 2023-12-01T17:44:55.169 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91857.log 2023-12-01T17:44:55.170 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.86994.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.86994.log.gz 2023-12-01T17:44:55.171 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92349.log 2023-12-01T17:44:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91857.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91857.log.gz 2023-12-01T17:44:55.173 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93154.log 2023-12-01T17:44:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92349.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.92349.log.gz 2023-12-01T17:44:55.179 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92878.log 2023-12-01T17:44:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93154.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.93154.log.gz 2023-12-01T17:44:55.185 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93339.log 2023-12-01T17:44:55.190 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92878.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.92878.log.gz 2023-12-01T17:44:55.191 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91941.log 2023-12-01T17:44:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.93339.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.93339.log.gz 2023-12-01T17:44:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159920.log 2023-12-01T17:44:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91941.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.91941.log.gz 2023-12-01T17:44:55.196 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92454.log 2023-12-01T17:44:55.197 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.159920.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159920.log.gz 2023-12-01T17:44:55.197 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90985.log 2023-12-01T17:44:55.207 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.92454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91259.log 2023-12-01T17:44:55.213 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90190.log 2023-12-01T17:44:55.214 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91259.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.90985.log.gz 2023-12-01T17:44:55.226 INFO:teuthology.orchestra.run.smithi067.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91259.log.gz 2023-12-01T17:44:55.228 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91561.log 2023-12-01T17:44:55.229 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.90190.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.90190.log.gz 2023-12-01T17:44:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87096.log 2023-12-01T17:44:55.236 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.91561.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.91561.log.gz 2023-12-01T17:44:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-12-01T17:44:55.249 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-client.admin.87096.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.87096.log.gz 2023-12-01T17:44:55.520 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/ceph-mon.c.log: 91.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-12-01T17:44:55.758 INFO:teuthology.orchestra.run.smithi067.stderr: 93.1% -- replaced with /var/log/ceph/ceph-client.admin.92454.log.gz 2023-12-01T17:44:56.360 INFO:teuthology.orchestra.run.smithi067.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.92440.log.gz 2023-12-01T17:44:57.211 INFO:teuthology.orchestra.run.smithi092.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-12-01T17:44:57.415 INFO:teuthology.orchestra.run.smithi067.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-12-01T17:44:57.939 INFO:teuthology.orchestra.run.smithi067.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-12-01T17:45:00.443 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-12-01T17:45:00.842 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-12-01T17:45:01.094 INFO:teuthology.orchestra.run.smithi067.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-12-01T17:45:01.196 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-12-01T17:45:01.381 INFO:teuthology.orchestra.run.smithi092.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-12-01T17:45:01.383 INFO:teuthology.orchestra.run.smithi092.stderr: 2023-12-01T17:45:01.383 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m6.490s 2023-12-01T17:45:01.383 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m25.660s 2023-12-01T17:45:01.383 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m0.903s 2023-12-01T17:45:01.790 INFO:teuthology.orchestra.run.smithi067.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-12-01T17:45:02.348 INFO:teuthology.orchestra.run.smithi067.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-12-01T17:45:02.614 INFO:teuthology.orchestra.run.smithi067.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-12-01T17:45:02.616 INFO:teuthology.orchestra.run.smithi067.stderr: 2023-12-01T17:45:02.616 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m7.754s 2023-12-01T17:45:02.616 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m34.953s 2023-12-01T17:45:02.616 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m1.158s 2023-12-01T17:45:02.616 INFO:tasks.ceph:Archiving logs... 2023-12-01T17:45:02.617 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/remote/smithi067/log 2023-12-01T17:45:02.618 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-01T17:45:07.547 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/remote/smithi092/log 2023-12-01T17:45:07.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-01T17:45:11.730 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-12-01T17:45:11.732 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-01T17:45:11.749 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa/tasks/ceph.py", line 1957, in task mon0_remote.run( File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d710c09d2ffe0f0858468330fd3d69f1f4506222/qa/tasks/ceph.py", line 368, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2023-12-01T17:45:11.750 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-01T17:45:11.750 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-01T17:45:11.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-01T17:45:11.838 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-01T17:45:11.847 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-01T17:45:11.847 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T17:45:11.850 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-01T17:45:11.864 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2023-12-01T17:45:11.866 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2023-12-01T17:45:11.869 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T17:45:11.869 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2023-12-01T17:45:11.870 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +1840us[+1864us] +/- 77ms 2023-12-01T17:45:11.870 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -328us[ -305us] +/- 61ms 2023-12-01T17:45:11.870 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 -1402us[-1379us] +/- 75ms 2023-12-01T17:45:11.870 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T17:45:11.871 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-01T17:45:11.871 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-12-01T17:45:11.871 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 58 +1867us[+1867us] +/- 77ms 2023-12-01T17:45:11.871 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -352us[ -355us] +/- 61ms 2023-12-01T17:45:11.871 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 -1419us[-1422us] +/- 75ms 2023-12-01T17:45:11.871 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-12-01T17:45:11.872 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-01T17:45:11.887 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-01T17:45:11.887 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-01T17:45:11.898 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-01T17:45:12.018 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-01T17:45:12.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T17:45:12.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T17:45:12.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-01T17:45:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T17:45:12.192 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-01T17:45:12.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-01T17:45:12.224 INFO:teuthology.task.internal:Duration was 3519.289462 seconds 2023-12-01T17:45:12.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-01T17:45:12.254 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-01T17:45:12.255 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-01T17:45:12.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-01T17:45:12.307 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T17:45:12.308 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-01T17:45:12.580 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-01T17:45:12.580 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2023-12-01T17:45:12.580 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-01T17:45:12.601 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2023-12-01T17:45:12.601 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-01T17:45:12.642 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-01T17:45:12.642 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-01T17:45:12.644 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-01T17:45:12.711 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-01T17:45:12.711 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-01T17:45:12.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-01T17:45:12.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-01T17:45:12.856 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-01T17:45:12.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-01T17:45:12.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-01T17:45:12.915 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-01T17:45:12.925 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-01T17:45:12.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-01T17:45:12.956 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2023-12-01T17:45:12.980 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2023-12-01T17:45:13.008 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-01T17:45:13.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T17:45:13.030 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-01T17:45:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-01T17:45:13.062 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-01T17:45:13.076 INFO:teuthology.task.internal:Transferring archived files... 2023-12-01T17:45:13.077 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/remote/smithi067 2023-12-01T17:45:13.077 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-01T17:45:13.121 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838/remote/smithi092 2023-12-01T17:45:13.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-01T17:45:13.156 INFO:teuthology.task.internal:Removing archive directory... 2023-12-01T17:45:13.157 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-01T17:45:13.163 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-01T17:45:13.207 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-01T17:45:13.222 INFO:teuthology.task.internal:Not uploading archives. 2023-12-01T17:45:13.222 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-01T17:45:13.235 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-01T17:45:13.235 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-01T17:45:13.238 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-01T17:45:13.250 INFO:teuthology.orchestra.run.smithi067.stdout: 265256 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 1 17:45 /home/ubuntu/cephtest 2023-12-01T17:45:13.263 INFO:teuthology.orchestra.run.smithi092.stdout: 265244 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 1 17:45 /home/ubuntu/cephtest 2023-12-01T17:45:13.263 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-01T17:45:13.281 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-01T17:45:13.295 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2023-12-01T17:45:13.295 INFO:teuthology.nuke:Checking targets against current locks 2023-12-01T17:45:13.326 DEBUG:teuthology.nuke:shortname: smithi067 2023-12-01T17:45:13.326 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T17:45:13.351 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 16:38:16.157771', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBQIyZoIHRylwtXjzYm+hE6V8vXmABu3fn7ivKqnzNMIizqP9FIVhl3+6s6v1db7RCg8qcA69TIPmuMquZ12nzg='} 2023-12-01T17:45:13.354 DEBUG:teuthology.nuke:shortname: smithi092 2023-12-01T17:45:13.355 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-01T17:45:13.380 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-01_15:09:25-rados-wip-neorados-learning-from-experience-distro-default-smithi/7473838', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-01 16:38:16.155820', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCsY9SWMXphNX4his3hpk3d0Ii08nhsOTfGtAq7TpsCy1q9isqLbgjSEt+no/DLci5rxXXU1Cwoatw1jkwPA7Qw='} 2023-12-01T17:45:13.408 INFO:teuthology.orchestra.console.smithi067:Power off 2023-12-01T17:45:13.408 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T17:45:13.432 INFO:teuthology.orchestra.console.smithi092:Power off 2023-12-01T17:45:13.433 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-01T17:45:13.444 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2023-12-01T17:45:13.444 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T17:45:13.464 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2023-12-01T17:45:13.464 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T17:45:13.501 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2023-12-01T17:45:13.517 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-12-01T17:45:17.502 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T17:45:17.519 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T17:45:17.617 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2023-12-01T17:45:17.631 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2023-12-01T17:45:21.621 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T17:45:21.634 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T17:45:21.737 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2023-12-01T17:45:21.757 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2023-12-01T17:45:21.857 INFO:teuthology.orchestra.console.smithi092:Power off completed 2023-12-01T17:45:25.740 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-01T17:45:25.856 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2023-12-01T17:45:25.957 INFO:teuthology.orchestra.console.smithi067:Power off completed 2023-12-01T17:45:26.061 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-bitmap rados tasks/rados_api_tests validater/valgrind} duration: 3519.2894616127014 failure_reason: saw valgrind issues flavor: default owner: scheduled_yuriw@teuthology status: fail success: false 2023-12-01T17:45:26.061 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-01T17:45:26.147 INFO:teuthology.run:FAIL